{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Recurrent Neural Networks\n",
    "(c) Deniz Yuret, 2019\n",
    "* Objectives: learn about RNNs, the RNN layer, compare with MLP on a tagging task.\n",
    "* Prerequisites: [MLP models](40.mlp.ipynb)\n",
    "* New functions: \n",
    "[RNN](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.RNN),\n",
    "[adam](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.adam)\n",
    "\n",
    "![image](https://github.com/denizyuret/Knet.jl/blob/master/docs/src/images/RNN-unrolled.png?raw=true)([image\n",
    "source](http://colah.github.io/posts/2015-08-Understanding-LSTMs))\n",
    "\n",
    "In this notebook we will see how to implement a recurrent neural network (RNN) in Knet. In RNNs, connections between units form a directed cycle, which allows them to keep a persistent state over time. This gives them the ability to process sequences of arbitrary length one element at a time, while keeping track of what happened at previous elements. One can view the current state of the RNN as a representation for the sequence processed so far.\n",
    "\n",
    "We will build a part-of-speech tagger using a large annotated corpus of English. We will represent words with numeric vectors appropriate as inputs to a neural network. These word vectors will be initialized randomly and learned during training just like other model parameters. We will compare three network architectures: (1) an MLP which tags each word independently of its neighbors, (2) a simple RNN that can represent the neighboring words to the left, (3) a bidirectional RNN that can represent both left and right contexts. As can be expected 1 < 2 < 3 in performance. More surprisingly, the three models are very similar to each other: we will see their model diagrams are identical except for the horizontal connections that carry information across the sequence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Setup display width, load packages, import symbols\n",
    "ENV[\"COLUMNS\"] = 72\n",
    "using Pkg; for p in (\"Knet\",\"Plots\"); haskey(Pkg.installed(),p) || Pkg.add(p); end\n",
    "using Random: shuffle!\n",
    "using Base.Iterators: flatten\n",
    "using Knet: Knet, AutoGrad, param, param0, mat, RNN, relu, Data, adam, progress, nll, zeroone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Brown Corpus\n",
    "To introduce recurrent neural networks (RNNs) we will train a part-of-speech tagger using the [Brown Corpus](https://en.wikipedia.org/wiki/Brown_Corpus). We will train three models: a MLP, a unidirectional RNN, a bidirectional RNN and observe significant performance differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Brown Corpus has 57340 sentences, 1161192 tokens, with a word vocabulary of 56057 and a tag vocabulary of 472.\n"
     ]
    }
   ],
   "source": [
    "include(Knet.dir(\"data/nltk.jl\"))\n",
    "(data,words,tags) = brown()\n",
    "println(\"The Brown Corpus has $(length(data)) sentences, $(sum(length(p[1]) for p in data)) tokens, with a word vocabulary of $(length(words)) and a tag vocabulary of $(length(tags)).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`data` is an array of `(w,t)` pairs each representing a sentence, where `w` is a sequence of word ids, and `t` is a sequence of tag ids. `words` and `tags` contain the strings for the ids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "57340-element Array{Tuple{Array{UInt16,1},Array{UInt16,1}},1}\n",
      "56057-element Array{String,1}\n",
      "472-element Array{String,1}\n"
     ]
    }
   ],
   "source": [
    "println.(summary.((data,words,tags)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is what the first sentence looks like with ids and with strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2×25 Array{Int64,2}:\n",
       " 15  5860  1296  5597  17468  60  …  14   9  85  10004  221  189  3\n",
       "  3    40    21    39     21  13     29  14  46      7   13    1  5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "2×25 Array{String,2}:\n",
       " \"The\"  \"Fulton\"  \"County\"  \"Grand\"  \"Jury\"   …  \"took\"  \"place\"  \".\"\n",
       " \"at\"   \"np-tl\"   \"nn-tl\"   \"jj-tl\"  \"nn-tl\"     \"vbd\"   \"nn\"     \".\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "(w,t) = first(data)\n",
    "display(permutedims(Int[w t]))\n",
    "display(permutedims([words[w] tags[t]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Chain of layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's define a chain of layers\n",
    "struct Chain\n",
    "    layers\n",
    "    Chain(layers...) = new(layers)\n",
    "end\n",
    "(c::Chain)(x) = (for l in c.layers; x = l(x); end; x)\n",
    "(c::Chain)(x,y) = nll(c(x),y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dense layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Redefine dense layer (See mlp.ipynb):\n",
    "struct Dense; w; b; f; end\n",
    "Dense(i::Int,o::Int,f=identity) = Dense(param(o,i), param0(o), f)\n",
    "(d::Dense)(x) = d.f.(d.w * mat(x,dims=1) .+ d.b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embeddings\n",
    "`data` has each sentence tokenized into an array of words and each word mapped to a `UInt16` id. To use these words as inputs to a neural network we further map each word to a Float32 vector. We will keep these vectors in the columns of a size (X,V) matrix where X is the embedding dimension and V is the vocabulary size. The vectors will be initialized randomly, and trained just like any other network parameter. Let's define an embedding layer for this purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "struct Embed; w; end\n",
    "Embed(vocabsize::Int,embedsize::Int) = Embed(param(embedsize,vocabsize))\n",
    "(e::Embed)(x) = e.w[:,x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This is what the words, word ids and embeddings for a sentence looks like: (note the identical id and embedding for the 2nd and 5th words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×7 Array{String,2}:\n",
       " \"Rapping\"  \"the\"  \"stick\"  \"against\"  \"the\"  \"desk\"  \".\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "1×7 Array{Int64,2}:\n",
       " 47900  1  3014  163  1  1719  3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "8×7 Array{Float32,2}:\n",
       "  0.00132076   -0.00257328   …  -0.00199834   0.00529953 \n",
       " -0.00303089   -0.00174273      -0.00522823  -0.000440273\n",
       " -0.00489826    0.0047967        0.00488125  -0.000466263\n",
       " -0.00131182    0.000914559     -0.00269184   0.00331362 \n",
       "  0.00249676   -0.000997588     -0.00463821  -0.00275833 \n",
       "  0.00529401   -0.00142283   …   0.00496875   0.00417025 \n",
       " -0.00490186   -0.00371554       0.00543425  -0.000565609\n",
       " -0.000910787   0.00240753       0.0050205    0.00276742 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embedlayer = Embed(length(words),8)\n",
    "(w,t) = data[52855]\n",
    "display(permutedims(words[w]))\n",
    "display(permutedims(Int.(w)))\n",
    "display(embedlayer(w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RNN layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "\\begin{verbatim}\n",
       "rnn = RNN(inputSize, hiddenSize; opts...)\n",
       "rnn(x; batchSizes) => y\n",
       "rnn.h, rnn.c  # hidden and cell states\n",
       "\\end{verbatim}\n",
       "\\texttt{RNN} returns a callable RNN object \\texttt{rnn}. Given a minibatch of sequences \\texttt{x}, \\texttt{rnn(x)} returns \\texttt{y}, the hidden states of the final layer for each time step. \\texttt{rnn.h} and \\texttt{rnn.c} fields can be used to set the initial hidden states and read the final hidden states of all layers.  Note that the final time step of \\texttt{y} always contains the final hidden state of the last layer, equivalent to \\texttt{rnn.h} for a single layer network.\n",
       "\n",
       "\\textbf{Dimensions:} The input \\texttt{x} can be 1, 2, or 3 dimensional and \\texttt{y} will have the same number of dimensions as \\texttt{x}. size(x)=(X,[B,T]) and size(y)=(H/2H,[B,T]) where X is inputSize, B is batchSize, T is seqLength, H is hiddenSize, 2H is for bidirectional RNNs. By default a 1-D \\texttt{x} represents a single instance for a single time step, a 2-D \\texttt{x} represents a single minibatch for a single time step, and a 3-D \\texttt{x} represents a sequence of identically sized minibatches for multiple time steps. The output \\texttt{y} gives the hidden state (of the final layer for multi-layer RNNs) for each time step. The fields \\texttt{rnn.h} and \\texttt{rnn.c} represent the hidden states of all layers in a single time step and have size (H,B,L/2L) where L is numLayers and 2L is for bidirectional RNNs.\n",
       "\n",
       "\\textbf{batchSizes:} If \\texttt{batchSizes=nothing} (default), all sequences in a minibatch are assumed to be the same length. If \\texttt{batchSizes} is an array of (non-increasing) integers, it gives us the batch size for each time step (allowing different sequences in the minibatch to have different lengths). In this case \\texttt{x} will typically be 2-D with the second dimension representing variable size batches for time steps. If \\texttt{batchSizes} is used, \\texttt{sum(batchSizes)} should equal \\texttt{length(x) ÷ size(x,1)}. When the batch size is different in every time step, hidden states will have size (H,B,L/2L) where B is always the size of the first (largest) minibatch.\n",
       "\n",
       "\\textbf{Hidden states:} The hidden and cell states are kept in \\texttt{rnn.h} and \\texttt{rnn.c} fields (the cell state is only used by LSTM). They can be initialized during construction using the \\texttt{h} and \\texttt{c} keyword arguments, or modified later by direct assignment. Valid values are \\texttt{nothing} (default), \\texttt{0}, or an array of the right type and size possibly wrapped in a \\texttt{Param}. If the value is \\texttt{nothing} the initial state is assumed to be zero and the final state is discarded keeping the value \\texttt{nothing}. If the value is \\texttt{0} the initial state is assumed to be zero and \\texttt{0} is replaced by the final state on return. If the value is a valid state, it is used as the initial state and is replaced by the final state on return.\n",
       "\n",
       "In a differentiation context the returned final hidden states will be wrapped in \\texttt{Result} types. This is necessary if the same RNN object is to be called multiple times in a single iteration. Between iterations (i.e. after diff/update) the hidden states need to be unboxed with e.g. \\texttt{rnn.h = value(rnn.h)} to prevent spurious dependencies. This happens automatically during the backward pass for GPU RNNs but needs to be done manually for CPU RNNs. See the \\href{https://github.com/denizyuret/Knet.jl/blob/master/tutorial/80.charlm.ipynb}{CharLM Tutorial} for an example.\n",
       "\n",
       "\\textbf{Keyword arguments for RNN:}\n",
       "\n",
       "\\begin{itemize}\n",
       "\\item \\texttt{h=nothing}: Initial hidden state.\n",
       "\n",
       "\n",
       "\\item \\texttt{c=nothing}: Initial cell state.\n",
       "\n",
       "\n",
       "\\item \\texttt{rnnType=:lstm} Type of RNN: One of :relu, :tanh, :lstm, :gru.\n",
       "\n",
       "\n",
       "\\item \\texttt{numLayers=1}: Number of RNN layers.\n",
       "\n",
       "\n",
       "\\item \\texttt{bidirectional=false}: Create a bidirectional RNN if \\texttt{true}.\n",
       "\n",
       "\n",
       "\\item \\texttt{dropout=0}: Dropout probability. Applied to input and between layers.\n",
       "\n",
       "\n",
       "\\item \\texttt{skipInput=false}: Do not multiply the input with a matrix if \\texttt{true}.\n",
       "\n",
       "\n",
       "\\item \\texttt{dataType=Float32}: Data type to use for weights.\n",
       "\n",
       "\n",
       "\\item \\texttt{algo=0}: Algorithm to use, see CUDNN docs for details.\n",
       "\n",
       "\n",
       "\\item \\texttt{seed=0}: Random number seed for dropout. Uses \\texttt{time()} if 0.\n",
       "\n",
       "\n",
       "\\item \\texttt{winit=xavier}: Weight initialization method for matrices.\n",
       "\n",
       "\n",
       "\\item \\texttt{binit=zeros}: Weight initialization method for bias vectors.\n",
       "\n",
       "\n",
       "\\item \\texttt{usegpu=(gpu()>=0)}: GPU used by default if one exists.\n",
       "\n",
       "\\end{itemize}\n",
       "\\textbf{Formulas:} RNNs compute the output h[t] for a given iteration from the recurrent input h[t-1] and the previous layer input x[t] given matrices W, R and biases bW, bR from the following equations:\n",
       "\n",
       "\\texttt{:relu} and \\texttt{:tanh}: Single gate RNN with activation function f:\n",
       "\n",
       "\\begin{verbatim}\n",
       "h[t] = f(W * x[t] .+ R * h[t-1] .+ bW .+ bR)\n",
       "\\end{verbatim}\n",
       "\\texttt{:gru}: Gated recurrent unit:\n",
       "\n",
       "\\begin{verbatim}\n",
       "i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\n",
       "r[t] = sigm(Wr * x[t] .+ Rr * h[t-1] .+ bWr .+ bRr) # reset gate\n",
       "n[t] = tanh(Wn * x[t] .+ r[t] .* (Rn * h[t-1] .+ bRn) .+ bWn) # new gate\n",
       "h[t] = (1 - i[t]) .* n[t] .+ i[t] .* h[t-1]\n",
       "\\end{verbatim}\n",
       "\\texttt{:lstm}: Long short term memory unit with no peephole connections:\n",
       "\n",
       "\\begin{verbatim}\n",
       "i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\n",
       "f[t] = sigm(Wf * x[t] .+ Rf * h[t-1] .+ bWf .+ bRf) # forget gate\n",
       "o[t] = sigm(Wo * x[t] .+ Ro * h[t-1] .+ bWo .+ bRo) # output gate\n",
       "n[t] = tanh(Wn * x[t] .+ Rn * h[t-1] .+ bWn .+ bRn) # new gate\n",
       "c[t] = f[t] .* c[t-1] .+ i[t] .* n[t]               # cell output\n",
       "h[t] = o[t] .* tanh(c[t])\n",
       "\\end{verbatim}\n"
      ],
      "text/markdown": [
       "```\n",
       "rnn = RNN(inputSize, hiddenSize; opts...)\n",
       "rnn(x; batchSizes) => y\n",
       "rnn.h, rnn.c  # hidden and cell states\n",
       "```\n",
       "\n",
       "`RNN` returns a callable RNN object `rnn`. Given a minibatch of sequences `x`, `rnn(x)` returns `y`, the hidden states of the final layer for each time step. `rnn.h` and `rnn.c` fields can be used to set the initial hidden states and read the final hidden states of all layers.  Note that the final time step of `y` always contains the final hidden state of the last layer, equivalent to `rnn.h` for a single layer network.\n",
       "\n",
       "**Dimensions:** The input `x` can be 1, 2, or 3 dimensional and `y` will have the same number of dimensions as `x`. size(x)=(X,[B,T]) and size(y)=(H/2H,[B,T]) where X is inputSize, B is batchSize, T is seqLength, H is hiddenSize, 2H is for bidirectional RNNs. By default a 1-D `x` represents a single instance for a single time step, a 2-D `x` represents a single minibatch for a single time step, and a 3-D `x` represents a sequence of identically sized minibatches for multiple time steps. The output `y` gives the hidden state (of the final layer for multi-layer RNNs) for each time step. The fields `rnn.h` and `rnn.c` represent the hidden states of all layers in a single time step and have size (H,B,L/2L) where L is numLayers and 2L is for bidirectional RNNs.\n",
       "\n",
       "**batchSizes:** If `batchSizes=nothing` (default), all sequences in a minibatch are assumed to be the same length. If `batchSizes` is an array of (non-increasing) integers, it gives us the batch size for each time step (allowing different sequences in the minibatch to have different lengths). In this case `x` will typically be 2-D with the second dimension representing variable size batches for time steps. If `batchSizes` is used, `sum(batchSizes)` should equal `length(x) ÷ size(x,1)`. When the batch size is different in every time step, hidden states will have size (H,B,L/2L) where B is always the size of the first (largest) minibatch.\n",
       "\n",
       "**Hidden states:** The hidden and cell states are kept in `rnn.h` and `rnn.c` fields (the cell state is only used by LSTM). They can be initialized during construction using the `h` and `c` keyword arguments, or modified later by direct assignment. Valid values are `nothing` (default), `0`, or an array of the right type and size possibly wrapped in a `Param`. If the value is `nothing` the initial state is assumed to be zero and the final state is discarded keeping the value `nothing`. If the value is `0` the initial state is assumed to be zero and `0` is replaced by the final state on return. If the value is a valid state, it is used as the initial state and is replaced by the final state on return.\n",
       "\n",
       "In a differentiation context the returned final hidden states will be wrapped in `Result` types. This is necessary if the same RNN object is to be called multiple times in a single iteration. Between iterations (i.e. after diff/update) the hidden states need to be unboxed with e.g. `rnn.h = value(rnn.h)` to prevent spurious dependencies. This happens automatically during the backward pass for GPU RNNs but needs to be done manually for CPU RNNs. See the [CharLM Tutorial](https://github.com/denizyuret/Knet.jl/blob/master/tutorial/80.charlm.ipynb) for an example.\n",
       "\n",
       "**Keyword arguments for RNN:**\n",
       "\n",
       "  * `h=nothing`: Initial hidden state.\n",
       "  * `c=nothing`: Initial cell state.\n",
       "  * `rnnType=:lstm` Type of RNN: One of :relu, :tanh, :lstm, :gru.\n",
       "  * `numLayers=1`: Number of RNN layers.\n",
       "  * `bidirectional=false`: Create a bidirectional RNN if `true`.\n",
       "  * `dropout=0`: Dropout probability. Applied to input and between layers.\n",
       "  * `skipInput=false`: Do not multiply the input with a matrix if `true`.\n",
       "  * `dataType=Float32`: Data type to use for weights.\n",
       "  * `algo=0`: Algorithm to use, see CUDNN docs for details.\n",
       "  * `seed=0`: Random number seed for dropout. Uses `time()` if 0.\n",
       "  * `winit=xavier`: Weight initialization method for matrices.\n",
       "  * `binit=zeros`: Weight initialization method for bias vectors.\n",
       "  * `usegpu=(gpu()>=0)`: GPU used by default if one exists.\n",
       "\n",
       "**Formulas:** RNNs compute the output h[t] for a given iteration from the recurrent input h[t-1] and the previous layer input x[t] given matrices W, R and biases bW, bR from the following equations:\n",
       "\n",
       "`:relu` and `:tanh`: Single gate RNN with activation function f:\n",
       "\n",
       "```\n",
       "h[t] = f(W * x[t] .+ R * h[t-1] .+ bW .+ bR)\n",
       "```\n",
       "\n",
       "`:gru`: Gated recurrent unit:\n",
       "\n",
       "```\n",
       "i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\n",
       "r[t] = sigm(Wr * x[t] .+ Rr * h[t-1] .+ bWr .+ bRr) # reset gate\n",
       "n[t] = tanh(Wn * x[t] .+ r[t] .* (Rn * h[t-1] .+ bRn) .+ bWn) # new gate\n",
       "h[t] = (1 - i[t]) .* n[t] .+ i[t] .* h[t-1]\n",
       "```\n",
       "\n",
       "`:lstm`: Long short term memory unit with no peephole connections:\n",
       "\n",
       "```\n",
       "i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\n",
       "f[t] = sigm(Wf * x[t] .+ Rf * h[t-1] .+ bWf .+ bRf) # forget gate\n",
       "o[t] = sigm(Wo * x[t] .+ Ro * h[t-1] .+ bWo .+ bRo) # output gate\n",
       "n[t] = tanh(Wn * x[t] .+ Rn * h[t-1] .+ bWn .+ bRn) # new gate\n",
       "c[t] = f[t] .* c[t-1] .+ i[t] .* n[t]               # cell output\n",
       "h[t] = o[t] .* tanh(c[t])\n",
       "```\n"
      ],
      "text/plain": [
       "\u001b[36m  rnn = RNN(inputSize, hiddenSize; opts...)\u001b[39m\n",
       "\u001b[36m  rnn(x; batchSizes) => y\u001b[39m\n",
       "\u001b[36m  rnn.h, rnn.c  # hidden and cell states\u001b[39m\n",
       "\n",
       "  \u001b[36mRNN\u001b[39m returns a callable RNN object \u001b[36mrnn\u001b[39m. Given a minibatch of\n",
       "  sequences \u001b[36mx\u001b[39m, \u001b[36mrnn(x)\u001b[39m returns \u001b[36my\u001b[39m, the hidden states of the final layer\n",
       "  for each time step. \u001b[36mrnn.h\u001b[39m and \u001b[36mrnn.c\u001b[39m fields can be used to set the\n",
       "  initial hidden states and read the final hidden states of all\n",
       "  layers. Note that the final time step of \u001b[36my\u001b[39m always contains the final\n",
       "  hidden state of the last layer, equivalent to \u001b[36mrnn.h\u001b[39m for a single\n",
       "  layer network.\n",
       "\n",
       "  \u001b[1mDimensions:\u001b[22m The input \u001b[36mx\u001b[39m can be 1, 2, or 3 dimensional and \u001b[36my\u001b[39m will\n",
       "  have the same number of dimensions as \u001b[36mx\u001b[39m. size(x)=(X,[B,T]) and\n",
       "  size(y)=(H/2H,[B,T]) where X is inputSize, B is batchSize, T is\n",
       "  seqLength, H is hiddenSize, 2H is for bidirectional RNNs. By default\n",
       "  a 1-D \u001b[36mx\u001b[39m represents a single instance for a single time step, a 2-D \u001b[36mx\u001b[39m\n",
       "  represents a single minibatch for a single time step, and a 3-D \u001b[36mx\u001b[39m\n",
       "  represents a sequence of identically sized minibatches for multiple\n",
       "  time steps. The output \u001b[36my\u001b[39m gives the hidden state (of the final layer\n",
       "  for multi-layer RNNs) for each time step. The fields \u001b[36mrnn.h\u001b[39m and \u001b[36mrnn.c\u001b[39m\n",
       "  represent the hidden states of all layers in a single time step and\n",
       "  have size (H,B,L/2L) where L is numLayers and 2L is for\n",
       "  bidirectional RNNs.\n",
       "\n",
       "  \u001b[1mbatchSizes:\u001b[22m If \u001b[36mbatchSizes=nothing\u001b[39m (default), all sequences in a\n",
       "  minibatch are assumed to be the same length. If \u001b[36mbatchSizes\u001b[39m is an\n",
       "  array of (non-increasing) integers, it gives us the batch size for\n",
       "  each time step (allowing different sequences in the minibatch to\n",
       "  have different lengths). In this case \u001b[36mx\u001b[39m will typically be 2-D with\n",
       "  the second dimension representing variable size batches for time\n",
       "  steps. If \u001b[36mbatchSizes\u001b[39m is used, \u001b[36msum(batchSizes)\u001b[39m should equal \u001b[36mlength(x)\n",
       "  ÷ size(x,1)\u001b[39m. When the batch size is different in every time step,\n",
       "  hidden states will have size (H,B,L/2L) where B is always the size\n",
       "  of the first (largest) minibatch.\n",
       "\n",
       "  \u001b[1mHidden states:\u001b[22m The hidden and cell states are kept in \u001b[36mrnn.h\u001b[39m and\n",
       "  \u001b[36mrnn.c\u001b[39m fields (the cell state is only used by LSTM). They can be\n",
       "  initialized during construction using the \u001b[36mh\u001b[39m and \u001b[36mc\u001b[39m keyword arguments,\n",
       "  or modified later by direct assignment. Valid values are \u001b[36mnothing\u001b[39m\n",
       "  (default), \u001b[36m0\u001b[39m, or an array of the right type and size possibly\n",
       "  wrapped in a \u001b[36mParam\u001b[39m. If the value is \u001b[36mnothing\u001b[39m the initial state is\n",
       "  assumed to be zero and the final state is discarded keeping the\n",
       "  value \u001b[36mnothing\u001b[39m. If the value is \u001b[36m0\u001b[39m the initial state is assumed to be\n",
       "  zero and \u001b[36m0\u001b[39m is replaced by the final state on return. If the value is\n",
       "  a valid state, it is used as the initial state and is replaced by\n",
       "  the final state on return.\n",
       "\n",
       "  In a differentiation context the returned final hidden states will\n",
       "  be wrapped in \u001b[36mResult\u001b[39m types. This is necessary if the same RNN object\n",
       "  is to be called multiple times in a single iteration. Between\n",
       "  iterations (i.e. after diff/update) the hidden states need to be\n",
       "  unboxed with e.g. \u001b[36mrnn.h = value(rnn.h)\u001b[39m to prevent spurious\n",
       "  dependencies. This happens automatically during the backward pass\n",
       "  for GPU RNNs but needs to be done manually for CPU RNNs. See the\n",
       "  CharLM Tutorial\n",
       "  (https://github.com/denizyuret/Knet.jl/blob/master/tutorial/80.charlm.ipynb)\n",
       "  for an example.\n",
       "\n",
       "  \u001b[1mKeyword arguments for RNN:\u001b[22m\n",
       "\n",
       "    •    \u001b[36mh=nothing\u001b[39m: Initial hidden state.\n",
       "\n",
       "    •    \u001b[36mc=nothing\u001b[39m: Initial cell state.\n",
       "\n",
       "    •    \u001b[36mrnnType=:lstm\u001b[39m Type of RNN: One of :relu, :tanh, :lstm,\n",
       "        :gru.\n",
       "\n",
       "    •    \u001b[36mnumLayers=1\u001b[39m: Number of RNN layers.\n",
       "\n",
       "    •    \u001b[36mbidirectional=false\u001b[39m: Create a bidirectional RNN if \u001b[36mtrue\u001b[39m.\n",
       "\n",
       "    •    \u001b[36mdropout=0\u001b[39m: Dropout probability. Applied to input and\n",
       "        between layers.\n",
       "\n",
       "    •    \u001b[36mskipInput=false\u001b[39m: Do not multiply the input with a matrix\n",
       "        if \u001b[36mtrue\u001b[39m.\n",
       "\n",
       "    •    \u001b[36mdataType=Float32\u001b[39m: Data type to use for weights.\n",
       "\n",
       "    •    \u001b[36malgo=0\u001b[39m: Algorithm to use, see CUDNN docs for details.\n",
       "\n",
       "    •    \u001b[36mseed=0\u001b[39m: Random number seed for dropout. Uses \u001b[36mtime()\u001b[39m if 0.\n",
       "\n",
       "    •    \u001b[36mwinit=xavier\u001b[39m: Weight initialization method for matrices.\n",
       "\n",
       "    •    \u001b[36mbinit=zeros\u001b[39m: Weight initialization method for bias\n",
       "        vectors.\n",
       "\n",
       "    •    \u001b[36musegpu=(gpu()>=0)\u001b[39m: GPU used by default if one exists.\n",
       "\n",
       "  \u001b[1mFormulas:\u001b[22m RNNs compute the output h[t] for a given iteration from\n",
       "  the recurrent input h[t-1] and the previous layer input x[t] given\n",
       "  matrices W, R and biases bW, bR from the following equations:\n",
       "\n",
       "  \u001b[36m:relu\u001b[39m and \u001b[36m:tanh\u001b[39m: Single gate RNN with activation function f:\n",
       "\n",
       "\u001b[36m  h[t] = f(W * x[t] .+ R * h[t-1] .+ bW .+ bR)\u001b[39m\n",
       "\n",
       "  \u001b[36m:gru\u001b[39m: Gated recurrent unit:\n",
       "\n",
       "\u001b[36m  i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\u001b[39m\n",
       "\u001b[36m  r[t] = sigm(Wr * x[t] .+ Rr * h[t-1] .+ bWr .+ bRr) # reset gate\u001b[39m\n",
       "\u001b[36m  n[t] = tanh(Wn * x[t] .+ r[t] .* (Rn * h[t-1] .+ bRn) .+ bWn) # new gate\u001b[39m\n",
       "\u001b[36m  h[t] = (1 - i[t]) .* n[t] .+ i[t] .* h[t-1]\u001b[39m\n",
       "\n",
       "  \u001b[36m:lstm\u001b[39m: Long short term memory unit with no peephole connections:\n",
       "\n",
       "\u001b[36m  i[t] = sigm(Wi * x[t] .+ Ri * h[t-1] .+ bWi .+ bRi) # input gate\u001b[39m\n",
       "\u001b[36m  f[t] = sigm(Wf * x[t] .+ Rf * h[t-1] .+ bWf .+ bRf) # forget gate\u001b[39m\n",
       "\u001b[36m  o[t] = sigm(Wo * x[t] .+ Ro * h[t-1] .+ bWo .+ bRo) # output gate\u001b[39m\n",
       "\u001b[36m  n[t] = tanh(Wn * x[t] .+ Rn * h[t-1] .+ bWn .+ bRn) # new gate\u001b[39m\n",
       "\u001b[36m  c[t] = f[t] .* c[t-1] .+ i[t] .* n[t]               # cell output\u001b[39m\n",
       "\u001b[36m  h[t] = o[t] .* tanh(c[t])\u001b[39m"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@doc RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The three taggers: MLP, RNN, biRNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tagger0 (MLP)\n",
    "This is what Tagger0 looks like. Every tag is predicted independently. The prediction of each tag only depends on the corresponding word.\n",
    "<img src=\"https://docs.google.com/drawings/d/e/2PACX-1vTfV4-TB0KwjDbFKpj3rL0tfeApEh9XXaDJ1OF3emNVAmc_-hvgqpEBuA_K0FsNuxymZrv3ztScXxqF/pub?w=378&h=336\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tagger1 (RNN) \n",
    "In Tagger1, the RNN layer takes its previous output as an additional input. The prediction of each tag is based on words to the left.\n",
    "<img src=\"https://docs.google.com/drawings/d/e/2PACX-1vTaizzCISuSxihPCjndr7xMVwklsrefi9zn7ZArCvsR8fb5V4DGKtusyIzn3Ujp3QbAJgUz1WSlLvIJ/pub?w=548&h=339\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tagger2 (biRNN)\n",
    "In Tagger2 there are two RNNs: the forward RNN reads the sequence from left to right, the backward RNN reads it from right to left. The prediction of each tag is dependent on all the words in the sentence.\n",
    "<img src=\"https://docs.google.com/drawings/d/e/2PACX-1vQawvnCj6odRF2oakF_TgXd8gLxSsfQP8-2ZdBdEIpfgIyPq0Zp_EF6zcFJf6JlGhfiKQvdVyg-Weq2/pub?w=566&h=335\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "Tagger0(vocab,embed,hidden,output)=  # MLP Tagger\n",
    "    Chain(Embed(vocab,embed),Dense(embed,hidden,relu),Dense(hidden,output))\n",
    "Tagger1(vocab,embed,hidden,output)=  # RNN Tagger\n",
    "    Chain(Embed(vocab,embed),RNN(embed,hidden,rnnType=:relu),Dense(hidden,output))\n",
    "Tagger2(vocab,embed,hidden,output)=  # biRNN Tagger\n",
    "    Chain(Embed(vocab,embed),RNN(embed,hidden,rnnType=:relu,bidirectional=true),Dense(2hidden,output));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequence Minibatching\n",
    "Minibatching is a bit more complicated with sequences compared to simple classification problems, this section can be skipped on a first reading. In addition to the input and minibatch sizes, there is also the time dimension to consider. To keep things simple we will concatenate all sentences into one big sequence, then split this sequence into equal sized chunks. The input to the tagger will be size (B,T) where B is the minibatch size, and T is the chunk size. The input to the RNN layer will be size (X,B,T) where X is the embedding size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "BATCHSIZE = 64\n",
    "SEQLENGTH = 32;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "function seqbatch(x,y,B,T)\n",
    "    N = length(x) ÷ B\n",
    "    x = permutedims(reshape(x[1:N*B],N,B))\n",
    "    y = permutedims(reshape(y[1:N*B],N,B))\n",
    "    d = []; for i in 0:T:N-T\n",
    "        push!(d, (x[:,i+1:i+T], y[:,i+1:i+T]))\n",
    "    end\n",
    "    return d\n",
    "end\n",
    "allw = vcat((x->x[1]).(data)...)\n",
    "allt = vcat((x->x[2]).(data)...)\n",
    "d = seqbatch(allw, allt, BATCHSIZE, SEQLENGTH);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This may be a bit more clear if we look at an example minibatch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64×32 Array{String,2}:\n",
       " \"The\"              \"Fulton\"    …  \"term-end\"     \"presentments\" \n",
       " \"director\"         \"of\"           \"gifts\"        \"from\"         \n",
       " \"local\"            \"club\"         \"who\"          \"will\"         \n",
       " \".\"                \"The\"          \"of\"           \"the\"          \n",
       " \"plans\"            \"titled\"       \"this\"         \"fall\"         \n",
       " \"admitted\"         \"for\"       …  \"''\"           \".\"            \n",
       " \"American\"         \"system\"       \"patriotic\"    \"apprehensions\"\n",
       " \"of\"               \"the\"          \"Then\"         \"cometh\"       \n",
       " \"the\"              \"misdeeds\"     \"is\"           \"entitled\"     \n",
       " \"entertaining\"     \"and\"          \"from\"         \"these\"        \n",
       " \"is\"               \"merely\"    …  \"to\"           \"the\"          \n",
       " \"is\"               \"not\"          \"that\"         \"any\"          \n",
       " \"has\"              \"said\"         \"important\"    \"that\"         \n",
       " ⋮                              ⋱  ⋮                             \n",
       " \".\"                \"But\"          \"would\"        \"I\"            \n",
       " \"not\"              \"worth\"        \"Constable's\"  \"explanation\"  \n",
       " \",\"                \"I\"            \"carries\"      \",\"            \n",
       " \"not\"              \"hurt\"      …  \"happy\"        \"to\"           \n",
       " \"you\"              \"do\"           \"away\"         \".\"            \n",
       " \"west\"             \",\"            \"Water\"        \"splashed\"     \n",
       " \"caught\"           \"sight\"        \"rapidly\"      \".\"            \n",
       " \"city\"             \".\"            \".\"            \"It\"           \n",
       " \"her\"              \"life\"      …  \"him\"          \"lead\"         \n",
       " \"dog\"              \".\"            \"that's\"       \"the\"          \n",
       " \"what's-his-name\"  \"got\"          \"spent\"        \"the\"          \n",
       " \"solid\"            \",\"            \"dislike\"      \"them\"         "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(x,y) = first(d)\n",
    "words[x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Embedding a minibatch\n",
    "Julia indexing allows us to get the embeddings for this minibatch in one go as an (X,B,T) array where X is the embedding size, B is the minibatch size, and T is the subsequence length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"128×64×32 Array{Float32,3}\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedlayer = Embed(length(words),128)\n",
    "summary(embedlayer(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(556, 10)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shuffle and split minibatches into train and test portions\n",
    "shuffle!(d)\n",
    "dtst = d[1:10]\n",
    "dtrn = d[11:end]\n",
    "length.((dtrn,dtst))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainresults (generic function with 1 method)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For running experiments we will use the Adam algorithm which typically converges faster than SGD.\n",
    "function trainresults(file,maker,savemodel)\n",
    "    if (print(\"Train from scratch? \"); readline()[1]=='y')\n",
    "        model = maker()\n",
    "        takeevery(n,itr) = (x for (i,x) in enumerate(itr) if i % n == 1)\n",
    "        results = ((nll(model,dtst), zeroone(model,dtst))\n",
    "                   for x in takeevery(100, progress(adam(model,repeat(dtrn,5)))))\n",
    "        results = reshape(collect(Float32,flatten(results)),(2,:))\n",
    "        Knet.save(file,\"model\",(savemodel ? model : nothing),\"results\",results)\n",
    "        Knet.gc() # To save gpu memory\n",
    "    else\n",
    "        isfile(file) || download(\"http://people.csail.mit.edu/deniz/models/tutorial/$file\",file)\n",
    "        model,results = Knet.load(file,\"model\",\"results\")\n",
    "    end\n",
    "    println(minimum(results,dims=2))\n",
    "    return model,results\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "VOCABSIZE = length(words)\n",
    "EMBEDSIZE = 128\n",
    "HIDDENSIZE = 128\n",
    "OUTPUTSIZE = length(tags);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? stdin> n\n",
      "Float32[0.322172; 0.0997559]\n"
     ]
    }
   ],
   "source": [
    "# 2.35e-01  100.00%┣┫ 2780/2780 [00:13/00:13, 216.36i/s] [0.295007; 0.0972656]\n",
    "t0maker() = Tagger0(VOCABSIZE,EMBEDSIZE,HIDDENSIZE,OUTPUTSIZE)\n",
    "(t0,r0) = trainresults(\"tagger113a.jld2\",t0maker,false);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? stdin> n\n",
      "Float32[0.246886; 0.0693848]\n"
     ]
    }
   ],
   "source": [
    "# 1.49e-01  100.00%┣┫ 2780/2780 [00:19/00:19, 142.58i/s] [0.21358; 0.0616211]\n",
    "t1maker() = Tagger1(VOCABSIZE,EMBEDSIZE,HIDDENSIZE,OUTPUTSIZE)\n",
    "(t1,r1) = trainresults(\"tagger113b.jld2\",t1maker,false);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? stdin> n\n",
      "Float32[0.191677; 0.0494141]\n"
     ]
    }
   ],
   "source": [
    "# 9.37e-02  100.00%┣┫ 2780/2780 [00:25/00:25, 109.77i/s] [0.156669; 0.044043]\n",
    "t2maker() = Tagger2(VOCABSIZE,EMBEDSIZE,HIDDENSIZE,OUTPUTSIZE)\n",
    "(t2,r2) = trainresults(\"tagger113c.jld2\",t2maker,true);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "using Plots; default(fmt=:png,ls=:auto,ymirror=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdeXwTZfoA8OedyeS+e5/hKhTKfUpBFLzwaCvqeizq6up6sniu4ArF9QZXFMXfuh71WnG9QFQQFbAouCJyFQoFCqW0paVXmjR3MvP+/giUkt5tkknb5/vx4yd533dmnoQ0T96Zd96XUEoBIYQQ6q8YsQNACCGExISJECGEUL+GiRAhhFC/hokQIYRQv4aJECGEUL+GiRAhhFC/hokQIYRQv4aJECGEUEiYzeasrCyj0ZidnW02m1s24Hk+PT29ecm0adPIGffcc0944sREiBBCKCSWLl1qMpkqKytTU1OXLVsWULtixYrMzMxDhw41lVBKi4qKysvLGxsbGxsbX3nllfDEiYkQIYRQSKxZs2bevHkymWzevHmrV68OqB09evTixYubl5w6dcrj8eTk5CQkJNx8881WqzU8cZJuT7EmCALP88GNJswq7ad0Uq2SU7TTpsYFlEKsAgCA8j6+rlISm9JU6/P5WJYlhIQ61F5BEARKKcuyYgcSKbxeL8dxYkcRKXieJ4QwDP74BgCglPI8L5FIxA6kyz744IO1a9cGFPp8vkceeeSiiy4KKFer1TU1NQqFwul0xsXFtZrYCDmbhvbs2fPwww8vX748NTX1oYce8ng8H3/8cSheRWAM3U6EPM97vd7gRhNma4rXj4sdNUCb0k6b14qYwxZ4bYoAAILT1rD0LuOTq5pqa2tr9Xp9b/w0h4Ldbud5XqvVih1IpKiqqoqPjxc7ikhhsVg4jlMqlWIHEhG8Xq/Vao2KihI7kC4rLCw8duxYQOGuXbt27NixcePGgHKVSlVXVyeXyx0OR0xMjN1ub7nD5omwucrKyoyMjPr6+mBF3o7uf4OzLNvbf/vfNPKaDttcNYBe9i0vl0sBAOTyBoaR8h5Gdfq7XiaTyeVyTIR+PM/zPC+Xy8UOJFL4Px5iRxEp3G43x3H4hvixLOt2u3vjuzFhwoQJEyYEFKpUqr1797ZsnJiYWFZWlpaWVlFRkZSU1OHOd+3a5XK5MjMzAUAqlcpksqDE3KF+fZri4wOrN5f+3H6b4XrCMXCw4fQPFklUgq+uMvShIYRQr5eVlZWXl0cpzcvLy8nJ8Rfm5+e31d5ut8+ZM+fgwYMej+fpp5+++uqrwxNnv06ENwyfM8t0fofNLk0m35adToRsdIKvFhMhQgh1LDc3t6CgICUlpbCwcNGiRf7CmTNnttV++vTpS5YsycrKSkpKMpvNS5cuDU+c/fqc3nHLCYfXOTImvf1ms5PJ6weEh0cxgD1ChBDqNL1ev27duoDCgCuCzZ8SQu6777777rsvHME10697hIQQCkKHzS5OYrZXU5sXAEASFc/XVYU8MoQQQuHSr3uEA3WpnWmmksDEGJJfSa9KJYqx58szzgt1YAghhMKmX/cIC2uLXtnx7860nJ3MbCgXAIBRqFmtIcRxIYQQCp9+3SPMiE7PiO7gAqHf5Skk+/uOT6IihBDqdfp1j7DBbXlrzwedaTnSQHgKhy3dnHwAIYRQxOrXiVDFqS4ecEEnG1925iYK839fdhXtDGVcCCGEwqdfJ0KOkbh5Tycbz04m/suEhOV8NRWhjAshhFD49OtECADflfzI005NHX5xEvPLKerwARsV78M7KBBCqK/o74nwgYl3saRTM6ZqORgXTbZUUkl0Io/31COEUF/R3xPhkp+XnrR1tnvnv4lCEhWPs6whhFCf0d8T4T/OX5Co7uxCOZenkG/LqSQ6wVdXBd1dvgohhFBE6e+J8LfKXeWNJzvZeLSR2L1wzC3XXnErFXr3osQIIYT8+nsiZAkrZaWdbEwALksmG8qp5sJrCNuv5yJACKE+o78nwgnxYwxyfefbX55y+iYKhBBCfUN/T4SrD63bcmJb59tfksT8VEldeFoUIYT6iv5+fu+aYVd2qb1eCqOMZEfB0fGuw6qpl4coKoRQ6Pz1r3/dunWruDFQSgVBYNlO3bsVZtOnT3/ttdeCsiuz2Xzrrbdu27Zt+vTp77//vsEQuGIBz/MZGRlFRUUB5fv3758yZYrdbg9KGB3q74lwX83B45YTWUMu6/wms5OZ38tt6SUbMREi1BsVFxffcccd06ZNEzuQSLRt27b169cHa29Lly41mUyff/75I488smzZsueff7557YoVK1atWnXo0KGArSwWy2233eZwOIIVRoe6nwh9Pp/L5QpiKKKIlhh0eo3NZuv8JhdEkb/vj72httJms9ntdo7jJJL+/nvCz+Fw8DzPMP39fHsTu93epY9W32a32yUSiSCIf4md5/nBgwePGzdO7EAiUWVlJc/z7Xxut2zZsnNn4GTLhw8fbjUdrFmzZu3atTKZbN68eTk5OQGJcPTo0YMHD87KympeSCm97bbbFi5c+Ic//KEHr6NrevQNTnv/vXQssNWuWoNM1/lNxhjoUWLkHTbqcdMzQhdhL4LvRgB8N5rzvxX4hvQK7fwzuVyuhoaGgEK73d7qJhUVFSaTCQBMJlNlZeA8JDNnzmy5ydKlSwcPHnzdddd1Oege6H4ilEgkGo0miKGIotFmL7aXjEke2aWtLknh7Ufj4j02tVqt0WiwR+jHMAzP833gUxEsNpsN340mgiBwHKdUKsUOBCLzylzkYFm2nc/ttddee+211wYUrl+/fuXKlS0bU0oJIf4HPN/xIMMff/xxw4YNP/zwQxdD7qn+fhYrUR1/7bCsjtuda3YyKeXicMZRhBBqR2JiYllZGQBUVFQkJSV12H7Tpk1btmyRSqX+9EkICc+wpv6eCG0e+0ObFnV1q8uSmR8kIzx4EwVCCLUtKysrLy+PUpqXl5eTk+MvzM/Pb6v9M8880/wKC6V0+vTpYYizvydCtVT18kXPdHUrowy2pV//e/SUUISEEEJ9Q25ubkFBQUpKSmFh4aJFp7scrV4aFBde3II1h9ddNfhSjuW6tJV/JYpRA0MUFEII9Xp6vX7dunUBhQHDatoamBPOcVX9vUcIAEZ54D2enXF5Cvm2DMe/IYRQr4eJEM5LmsgyXR5FNjGaDKjeUx6meQ8QQgiFCiZCeH3nOyesFV3diiGwrHT59hN4uzRCKAgIIX/84x+bl8ydO9c/eNJf2+omEolEIpHIZLLJkyfv2bMnHIH2RZgI4eHJ9w7QpXRjQ8GYcOhEbdDjQQj1T/n5+W632//Y4/G0M7qyic/n8/l8VVVV2dnZd911V2jj67swEcLaI9/urS7sxobG+ARzdY1X/BmjEEJ9wfTp05vuJd+8eXPnZ0M1GAwPPPDA/v37QxZaH4eJEGakTE0zDOrGhqrY+LGk6qsTOGQGIRQE11xzzRdffOF/vGbNmjlz5nRyw9ra2uXLlw8fPjxkofVxePsEuHmP0+dScoqubiiJSpgh+eX2vXD9EGjl/D1CqJc4YqF/+y1M53aG6mDZ5NZH51155ZWPPPKI1+tlWXbDhg3Lli3rcG9N1w5Hjhz5wQcfBDPQ/gQTIZxsrPIJvkR1fFc3lEQnxLmqOAa+LhWyTdi3Rqi3ipaT24aG6desUdbmgTQazYQJE/Lz85VK5ciRI3W6jhcDwEnMgwITIYyPH929DbnUodxNjy10wz92C1kmBjuFCPVSBhlcHRm/Zf1nR9VqdefPi6Kei4h/e3HtrNr7xu73urEhYSVEpswxEZ8A35Xj7zKEUE9lZ2d/8803X331VXZ2ttix9CPYI4QJ8WMmxI/p9uYE4O9jmX/s4mcn45uJEOoRo9E4YsQIt9sdGxsbUNV8uTefzxfeuPo47BGC2dXw7bFN3d6cej1/GMjYvLDpJHYKEULd1HS17/vvv9+yZUtAIaXU10xALeohTIQgZaVRiu5MNwoAwvFC8ztPMgQWjGGe3o3LMiGEUO+DiRCUnGJ41NDubcskDvaWHaEe102DmUoH/FSFP9AQQqiXwUQIPoF/Ztvybm4slXOJA90lB1gCC8Ywz2CnECGEzjCbzVlZWUajMTs722w2t2zA83x6enrzkg0bNowYMUKv148YMeL7778PT5yYCIFjJEtn5nZ7c2naWPfhPQBwyxCm2ArbTmGnECGEAACWLl1qMpkqKytTU1Nbzg+wYsWKzMzMQ4cONZUIgjB37tzXXnutvr7+qaeeuv3228MTZ/cHOgqC4PV6gxiKiF7d9datGTfoZdqubujxeKQDMlzr35VfOhcAHskgz+wSvpzVT3Oh2+3meb5p1mDk8Xjw3WjidrsFQWDZLi95FnQ4xqR9giC087ktKSkpLy8PKNyzZ0+rA1nXrFmzdu1amUw2b968nJyc559/vnnt6NGjBw8enJWV1VTi8/k+/PDDWbNm2Ww2mUym1+t79lI6q/uJ0Ofz2e19ZDm+HNNs1svYfV1+OXa7nTMm+mpO2mqqiFJzXTwsLVBvLXeNM/THc6QOh4Pn+Uj4posQDoejz/yN9JzD4eA4LhKSEM9H4p8nIaTpb4dl2VGjRr3++utTpkzxVy1ZsuTJJ59s3tj/TrZT1W2CILTzuf3666+//PLLgEKz2RwdHd2ycUVFhclkAgB/vzCgdubMmQElUqn0iiuusNlsWq2WELJ169buvIBuoIjSorojZldDNzasrq72er2ekyWCz+sveXU/f/X3vqBG12vYbDaLxSJ2FBHk5MmTYocQQRoaGux2u9hRUErp7Nmz169fL3YUgZp/GzudzhdffDEjI6OpSq1WFxQUtGzcTlX3rFu37vLLLw/WVkql0ul0UkrtdrtSqWx121YDttlszz777MSJE7saSffgNUIAgGJzicPr7PbmXMIAwp7uW/8lndlRS/fWi/+zFyHUS8nl8nvuuefEiRNNJQsXLrz99ttbPf3YTpXoEhMTy8rKAKCioiIpKanD9sePH//b3/4GACqV6o477jh48GDIQwQAHCzjd+XgS7ox6Xar5Cw8PJJ5djeuUogQ6iaHw/HWW2/NmjWrqeSxxx7jef6ll15q2bidKtFlZWXl5eVRSvPy8nJycvyF7Sw4nJiY+M4772zZsoVS+sknn4wbNy48ceKsYAAAqw58kaCKm2ma3qO9CAIwDADcM5xZVuDdb2ZGGnAiboR6Ab6+ujH/i4BCzUXXs7oo/+PGTZ/ylrrmtbJBIxVjz/c/dh8rdO756ZyNCaOfc/fpx5Q2rHmjqYY1xmkuvKbVMJrWVAIAhmGaXyHjOC4vL2/GjBk5OTkB9xu0UyW63NzcuXPnpqSkjB8//sMPP/QXzpw5k7ZxFVMqla5Zs+bhhx8uKSlJT0/Py8sLT5yYCAEA/jji2h7uoeHzlVxKmmrKZQCglMCDI9kX9gr/uRCHjSDUG3CcJDoxoIxIuKbHrCGWcLLmtYz67BpJjEIVuDk550dw81pW0+Y8Vk3pweVy/etf/5o3b97OnTubaseNG/fAAw/ccccdP/30U8CG7VSJS6/Xr1u3LqAwIAsGPL3ggguav+rwwEQIAFBUd8TqbpycOL7be+CS09yH9/gTIQD8NYMZ/In3sIUZqsNOIUKRjtUY1DNy2mmgHH9hO7VcwgAuYUCb1YS0v/OW5HL5vffeu3jx4oDyxYsXr169euXKlS03aacKdQgTIQCARqqWsdKe7EE+bJzlm3eBUv8vQZUE7h/BvrBXyJuBnUKEUJfJ5XKnM3AEn0wmy8vLu+SSS1q2b6cKdQgHywAAxKlioxTGnuyBNcQycqW3qrSpZH4G880JoaQRh48ihLojLi6uZeF555139913tyxvvwq1DxMhAECx+dgXh77p4U5kw8a5D+1qeqqTwt3DmRf24vBRhFDHWo4fOXnyZKtV//znP2mz5ZnaqkKdh4kQACA9Ku320Tf1cCfyoWNdh/c0L3loJPtFiVBqw88lQghFLkyEAAD1TvOL23t6kVmePlF7+S3NS4wyuDOd+WcBdgoRQihyYSIEANDJtX8Ze0vH7dpFZAppSlpA4aOj2I+PChV27BQihFCEwkQIAMAStrDmUMftui5aDremMS/tw04hQghFKEyEpx1tKBFoSNLVY2PYD44I1d2fyhQhhFAIYSI87daRNxDS45vfBaHqmdup+5ykF6+AmwYzy/dH4sovCCGEMBGe9sH+T34p39HTvTAMa4hzH90XULxgDPNWkVDj6unuEUJ9Vas/xJsKCSGSM2Qy2cSJE7dv395U1Xw9woCt2qpCzWEiPO3WkTdMS57c8/20vIkCAJJV5LqBzKuF2ClECHXBggULmh77zrBYLDfeeOMdd9zRVPXSSy/t2xf4+7vDKtQEE+FpHt7zyo5/93w/AbfVN3liLPPGQcHs7vkREEL9xQsvvNCysM+sVhg5MBGeJmWlWWmXBWE/yWm8pY63mgPKU9UkK5VZeQCHjyIUceqd5p1VewHg+5J8ACg2l5Q0lFKgG49vAYAdlbsb3JZgtWknjCVLlsTExGRmZhYXF/tLWj2T2WdWK4wcmAjPUnOqE9aKnu6FYWRDRrmPBJ4dBYAnxjIrD/CN3p4eASEUXF7BZ/c6AMDitgKAy+d28W5KTz+1eey8IASrTTthCIJQWVl54YUXzp8/v2UtOUOlUj366KPNz5r6lyR85plnioqKArZqpyoMzGZzVlaW0WjMzs42mwO7BwDA83zAGopr164dOXKkXq+fMWPG4cOHwxQoRWf8XrlnZ+XeLm1SXV3t9XoDCj3lR711Va22n/uj7/k9fDfji3g2m81isYgdRQQ5efKk2CFEkIaGBrvdLnYUlFI6e/bs9evXix1FIAA4evQopbSmpkar1TYVBjyglDqdzuXLl48fPz6g6oknnsjMzPSfBe2wqi3r1q27/PLLuxp8W1stWLDg/vvvd7lc999//8KFCwNqX3nllcmTJzcPqbS0VK1W//LLLw6H48UXX8zMzOxqJN2DPcKzJsSPGRc/quf74ZIGSYytTBsPAE+MZV7ez9uwU4gQagPDMBzHtdPAv1rhoUOBc4AsXrzYbDa3tVphW1VdxfO8uwWvt/UvtTVr1sybN08mk82bN2/16tUBtaNHjw5Yc/HYsWM33njj1KlTFQrFn/70p5avMUS6vx6hy+Vqtavbqz2z++VHRt2rkMg72b6urs7lckkknX0b9QDnGbT//M1+96A+eIO9w+Hged5ut4sdSKQ4deqU2CFEEKvVynGcQqEQOxDweDxih9C69957Lzc3d/ny5TNnzmy/ZUhXK/R4PJWVlW3VPvPMM2+99VZAIaV00qRJLRtXVFSYTCYAMJlMLffZ8mVeeOGFF154IQDwPJ+bm3vDDTd0JuCe634ilMvlCQkJQQwlEryesKxL7SUSicFg6HwiBIBnZHT2Bn7BFL2izy2KbLfbeZ7XarViBxJB+t7fSLcplUqO45RKpdiBgFTao1W4Q4fjuISEhIyMjA8//LDDxu2sVtjq0Jh2qgJIpdJ2Prevv/7666+/HlC4fv36VrublFL/eB9KKc939v6xjRs3PvbYY5deeukzzzzTyU16CE+NnqOo7shXRzb0fD/u4gLzxy+3WjXKSK5IITPX+8pwJm6E0BmU0sWLF9fU1OTn56ekpDQVBjxo0itWK0xMTCwrKwOAioqKpKSkDttTSh9//PGnnnrqv//97wsvvNClPkZPYCI8R4I6LjMYt9VLYlOc+7ZBGyPE3jqf/fNQZspa33flmAsRQn1WVlZWXl4epTQvLy8nJ8dfmJ+f31b7X375Zc2aNV999VViYqLNZrPZbOGJExPhOXQybbW9xuHt6QU8VmtgtVGeiuK2GtyVznwyS3LHz/zCHbyA2RAh1Bfl5uYWFBSkpKQUFhYuWrTIX9jOFdD8/PxDhw4ZDAbNGeGJExNhoKK6YqcvCLOCyoaNcx/a3U6D8+PJ7jmSnbX0km99uDYFQqjv0ev169atKy8v/+qrr3Q6nb8w4PRs86dPPPFEwI0N4YkTE2Gga4ZdaVToe76fVicdDRAjhw2zJefHk8lrfb9WY8cQIYREgIkwULG55J/bA8dEdYNsyBhPaRH1djBQmyXw5Hj2tUxmzg++FftxAjaEEAo3TISBhhgG/m3KvJ7vh8gUxrmPQucW+81KZX7NkXx0VLg5n7fj7LgIIRRGmAhbsfbIt4fq2hzn0nmK0dOItLP35pvU5KerJAoWJn7pO9CAp0kRQihMMBG2YmL8WJMuOfzHlbPw1vns42OYmet8nx7D06QIIRQOfW52k2CIUUYV1R0ZHZshytFvTWNGGcl1G/lNJ+nKTJbD3yoIBdvWrVtxLsBW7d7d3lj3vgoTYSu8gm9fzcGgJELrtx9IU4fJM6Z0aatxUWT3NZLbt/Cz1vs+mcUmKltZkwwh1D2LFy9evnx52CZ0bhWl1Ov1RuZkb9OnTxc7hHDDRNgKFaecm3EdBUqgpxmIyFWugzu6mggBQMvB5xezr+4XJn7p++ACycVJmAsRCo7MzMzMzExxY/B6vRaLJTo6WtwwkB+ed2vdp0VrN5du7fl+5EPHutq9rb4dBOCBkczHMyW3/cQ/uQsnoEEIoZDARNi669NzLjKd3/P9cImDqMvB11d3ew8XJJBfs9kfKug1G/naIMx4gxBC6ByYCFsnUGH5b/8Sej7BDyGytNGuIx1MMdO+ZBXJv1KSpoX0z7xLdvKWCF1MDSGEeiVMhK1jCHP10CtIMC7MyYaOc3c011qHOAZenMLumiOpcsLQz7xP7uIbcZl7hBAKBkyEbdJK1aWW8p7vRz58kmLMtJ7vBwBS1eTf09ktV0qOWSHtU+/SvYKrs0tdIoQQah0mwjZV2Kqq7TU93w+ri1KMDk4i9EvXkw8uZDdeIdlZS4d95nuzSPDhzfcIIdRdmAjbNCY2Y1LiOLGjaNNIA/n0IvaTWeynx4S0z3xvFgk8DitFCEUSs9mclZVlNBqzs7PNZnPLBjzPp6end6YwpDARtufx/KfNLovYUbTnvFiy8QrJhxewHxULY1b7PivBmywQQpFi6dKlJpOpsrIyNTV12bJlAbUrVqzIzMwMmNmg1cJQw0TYnhcuzDXIdT3fD+V9tf9eRPlQrSsxPZ5suUryynns83uEqV/5vj6Bp0oRQuJbs2bNvHnzZDLZvHnzVq9eHVA7evToxYsXd6Yw1Ei3lwD2eDxWqzW40USaUlt5YX3RFakXt9Wgrq5Op9NJJB1P0ON++wnp7NtIclpQAwxEAb4+yT13QB4lFZ4Y4c6MDuuSTg6Hg+d5jUYTzoNGslOnTsXFxYkdRaSwWq0cxykUCrEDiQher7exsdFoNIodSJe98cYbn376aUBhY2NjcnLyzz//HFCuVqtramoUCoXT6YyLi2s1ZRDSShpqtTB0uj/FmkQi0emC0FuKZAMVbLwhTqds82V6PJ5OJkJb+gQ4eUSdMTGoAbbiFh3MTYc1J5gHf5cM0sJz42FcVKiPeZpEIuF5XqvVhul4Ec/hcPT5v5Eu4ThOqVSKHUVE8Hq9ANAbPx633HLLpZdeGlC4bdu2DRs2tGxMKSWE+B/wfOSOce9+ImQYhmH6+JlVI2c4VFespRqNVN1qA47jOI7rTCJUDBvfuPET7opbgx1j624cAnMGwptFwtWb+eF6cs0A5moTSVKFdsJSjuMYhuE4LqRH6UX8Hw+xo4gU3BliBxIpeum7YTKZTCZTQGFtbe0PP/zQsnFiYmJZWVlaWlpFRUVSUlJYAuyOPp7Jeu5QfbHD6+j5fmSDR3nKi6nb2fNddfaILPw1gzl2Azc/g/mtho5Z7Zv6le/FAuGoFcfTIITCISsrKy8vj1Kal5eXk5PjL8zPzxc1qFbg6hMdyE6bTSEImYNIZdKUNPex/fLhk3q+t86Ts5BjYnJMwFP2f6foZyXCjG8EvRSyTOSqFGZafFAmz0EIoVbk5ubOnTs3JSVl/PjxH374ob9w5syZ4bz+1xmYCDtQZq14f98ni6Y93PNdaWb9gdWKdm2cJTA9nkyPZ18+D3bX0a9PCHf8zHsEyE4lfxjIZMYRBlMiQiio9Hr9unXrAgoDsmCrSTHMmRITYQdStElByYIAIB8R1r5gWxgCE6LJhGj2yfFQaKaflQgP/sqfsNHZycwfBpHZyQyH58sRQv0Jfud17Ntjm/bVHBA7ipDIMJAnx7O/Xy3ZmiUZYSBP7xYSV3n//BP/9QmcxRQh1F9gj7BjY2Iz9LLeN8q5S4bqyMIxZOEYpsxO1xyny/cJc3/kp8WRS5OZS5NIhgFPmyKE+izsEXYsRhl92Hw0KLvyHD/YuPGToOwqRFJUZH4G8+OVksq53COj2FNOev0mPv4j7/Wb+DeLhMogjJ9FCKHIgomwEyjdc2pfUPZE5CrbL4GXjiOTSgIXJ5EXJrGF10l+yZZcnEQ2VtCRX3gnfulbuIPfWEE9OI8bQqhPwFOjHeNY7rZRN1GgBHp6hpCLTwVB8NVWSqITghJbeAzSkLvSyV3pwFN2Tx3dWEGf3MUX1NMpseTiRCbLREbo8dwpQqi3wkTYKWuPfMsx3BWD25x0tPNkaWPch3f3rkTYhD094pQsGMPUuGBjhfBdOV1RyKsk5LJkMiOKTI4iOMEaQqh3wUTYKTlplwdrV/K0sc4Dv6kyrwjWDsUSI4ebBjM3DQYAtqCefl9B/32E/cuvEgnjHaIlaTqSpiVpOkjTkiFaYpCJHS4SFU/h11rWJjDxWmqUgVFGDDKQ4pUZFBkwEXYKBfrKjn/Pm3Anx/T0HZMNG9+w9i2gFPrQpC6jjWS0kdw7yMfzvFuqPWKhR6z0iIV+VQpHrEKxhUpZGKIlaVoyVEeGaCFNR4ZoiU4a7jgbvXDCRuOVJAoTc1hYPPBdufD1CfptmZCk4BKVYPHxZg/Uu2m9G+QsGKTEKAOjDAwyYpSBQQZG2TklRhkkKgne24pCChNhpxAgOWmXs4Tt+a5YXZT6gjnU4yKyvrkeTYwcYuQkM+6cNH/KCU3ZcfVxOGIViq1UKYE0LUnTkWE6kmGADAMZoA7yBDduHvbU0d9r6Y4auqOGnpHthF0AACAASURBVLDTFBWpclI3D6lqkqyCZBVJVUGKmiSrSIoKUtRE2/umQY44xxrp16X0mzJhezWdHk+yU5nnJkm0PivHcUrl2d8gjV6od1OzG+rdYHbTejeYPVDnosca/SVCvRvMbqhx0XQ9GR9FxkWR8dFktJEo8XsLBRV+oDpLL9OWWk4M1AdOu94N2ktv6vlOepc4BcQpyPT4c7JcpQMOW2ixlR620DcO0gMNUOOkw/VkpJGM0JNRRjJcDyZ11xIjT6HQfDbzHWyg6XoyKZqcH08eGsVk6ImEAQBw+KDURsvtUG6nJ2zwazUttwtlNjhhpwxAipqkqiFZRZJVxKSGZBVJUgJPof1JBiweEM6dFspcz42Q02SVCH3f8OMp/FpNvz4hfHOC1rroVanM/SOYLy9hVGe+YyyWwE00HGg4Yjq9skub/9BOHxTU0911dFcdzTssHGygAzWnk+K4KDIuql+8vSikMBF2VpW9ut7VEJREiPwSlJCgJBcknP0GbPTCATPdb6YHGujGfUJhAzR66AgDGWkgGaf/g0TlOd+YFKDYQnfU0t9r6I5aureOJqvIpBgyMZr8KY0ZG0XkrXXjlRIYrifD9dDy+9figTI7PWGDcjstt9PNJ6HcLpx0gISArN0zAnpp4Nlup0tpPsSX26mPQqqaJCggWUWSVJCoJKlqSFCSZBXEKQjbm8+RN3rhu3LhmxN0fZmQqCRXpZJ3zmcnxQSzZ6+QwJRYMiX29B69AhSa6e46uquWflEi7K2ncQoy/kxeHB9NYuRBO3TQNXrhiIUWW2mRGSqt8nEJgv+DrcdcLqqwrgLcBwTcRFFTU2MwGDqzHmFr+6INa9/SXnw9o9YHLT5R2e32oC/Ma3ZDoZkWNtBCMy000/1m6hVgpIGM0BODDHbW0h01VC8jk6LJpBgyKYaMj46gc5uVlZUJCQkAYPfBCRutdPiTK1Q66Ak7VDpouZ3WuSBGQVJUkKgkySpIUpFEJeikoJIQnRQ0HKi5048jyvFG+k0Z/bpU+LWaTo0j2anMlamk/e67xWIJxcK8AoXDFrqrju6qPZ0dNRwZF02mxZHsVJIu3o09dt/pnHfECsUWesRKD1tooxfStGSIjgxWCwrBVeFT7jPTA2aqlZIMPfjPhYw0kBEGoomYj3FnrF+/fuXKlevXrxc7kG7CHmEXLPl56X3jb49TxQZnd4QwCnXN/z0e89cXGUXrC/8ig8y/aMbZr7NaF+wz04NmWu+GB0cyk2Iiugfgp2q7A+oToMpJy+xw0k7L7VDhoAX1YPFQuxesXrB6weY9/bh5UtRyoDr3sYYjg7WQoSeDtCHpYp5ywp46uqeO7q6je+qo2UOvTGHuHs58cTGjFvUrmyGQrifpevLHwadLjjXS3bX0x0p66beCQgI5JpKdykyNC2HP2+mDI1ZabKVHLFBs9V8Lh3o39Q8KS9PCebHkljRmiBaSzyyO7fUKFos7Olrjf1pqowfMsN9Mf66ibxwUDjbQGAUZoQd/UhxpIMP1eGU0hLBH2CM96hECAEDDl296y4uj736GcBH2m7/rQtEj7NWaeoRB0TwpWjxg953z2OqBI1Z6wAyVTpqmJcP1ZISBDNfDCD1J05Gu3qggUCi2nk17e+upm4dxUWTsmf9G6Lt88jNEPcL27aqlX50Qviql5XZ6VSqTbSKXJjE9zyj+S9G/Vp/+r6SRDtL4bxmCITqSpiVDtJCibm9cuNfrtVgs0dHRrdYKFI7b6P56eqAB9pvpATMtstBEJckwkDQt+K9zazgiaXYAOQuKZq+LIdD8LIKMgYEaMlgbqmzaVo/QbDbfeuut27Ztmz59+vvvv28wGAIa8DyfkZFRVFTU+U1CARNhF5ywVvxS8duNw+c0lfQ8EQKl9R8vp47GqD8vBiYIo1JFhIkwQHATYSc5fVBkoQcbaKGZFjVAoZmesFOTmozQk+F6yDCQdD1J1xGFJHCr/ebTaW9PHd1nprHyprQHY4wktYujlloSJRE2OWGjX52gX5UK26vpBQlMtolkpTJxXRm4Xe2E7TXC9mr6v2r6ew1NVJHzYk//N9LQ5e5m+4mwJZ7CMSvdZ6bHGoEXAACsXso3+/J28eD0ndPe6jn71MnTo1Y41kijZGSwFgZryRAtGayBwVoyWEuMPb6bqK1EuHDhQpvN9tJLLz3yyCMajeb5559vXrtixYpVq1b99ttvzdNQ+5uECCbCLrB7HVZ3Y4I6rqkkCIkQgPK+ureWsMY4w/XzexyjmDARBhAlEbbkEeCIhR5ooAcb4ICZHmygR6w0UUlG6EmaDqocsKeOlthouu505htjJGNDMBRT3ER4NgwPbCgXviyl35cLQ3Ukx8RktzFHoFeAvfVnu311LnpeLJkSS86LZabE9HSOiK4mwqCgAOV2etQKR6202EqPWuFoIz1qpQyBwRoyRHc2NQ7WQJKqC7m9rUQ4bNiwtWvXpqenFxUV5eTkHDp0qHntjz/+aLfbs7Kymqeh9jcJke4nQq/X63D0u8UISqwnouQGrfT0mf3a2lq9Xt/DRAgA4HX7jh+QpI3r6X5E5XA4eJ7XaDRiBxIpqqqq4uPjxY6iFT4BSu2kqJEpbmRiZHSUXkjXCqG+ad1qtXIcp1BEyu2zXgG21bDfVrLrK1iOgSsSfVckCalKYVc981sd83s9s6+BGaCik6KEyVHCBCM/VEuDeJHR6/VardaoqKjg7bL76tykxEaO2UiJnSk583+rFx5K9y0Y4Q1o/PXXX+fn5wcUlpWVud3uluVqtbqmpkahUDidzri4OKvV2vLohJyThjqzSdB1/xucYRiO61UDm4KhxHZCLVM1vXCO4ziOC0Ii5DhuxOSe7kRsEomkf34q2uL/eIgdRSs4gHQZpBv9zwgACxDy0/L+v5TIeUM4gEuS4ZJkunySr8BM1lWwT+yVVDhgYhRMjqZLxtLxBt+ZcUAkFOMKI+fjEc9BvBqmnn5GAXgAsHnBLZCWESYmJo4cOTKgkGXZ4uLilnumlBJC/A94vlOLfXdjk57r/r8uy7Kin+UIvznDr2x+B4VCoVAqlUFIhM1Q3kfYXjk+zP/B7Yefirb4Px5iRxEpvF5vJJwabdV5SjgvCZ4O429Rr9fr9Xoj893wayuyWbNmzZo1K6DQf2q0ZePExMSysrK0tLSKioqkpKTOHLcbm/QcTuHXNTWO2tyfXgjhASitee1R14EdITwEQgiFRVZWVl5eHqU0Ly8vJyfHX9jyDGqHm4QaJsKuiVFGPz3j8RAegBD9tffVr3rJfWx/CI+CEEKhl5ubW1BQkJKSUlhYuGjRIn/hzJkzu7pJqPXKU3Di2lT6s16mnRA/JkT7l6YMjfrTwrq8Z2LufZZLGtzxBgghFJH0ev26desCCgNGaAY8bXWTUMMeYZeNiBo6zDgkpIeQpY013DC/9q0nffWnQnoghBBCmAi7LE4VU2w+FuqjKEZlai6+wbzqpVAfCCGE+jk8NdodO6sKxsSNJG0vHBMU6ulXKSe2dzIdIYRQz2GPsMsYwtwxZm6YjiVXhedACCHUb2Ei7I4NxzavPfxtOI/oOV4EghDOIyKEUD+Bp0a7Y/agwPtJQ836/UesPsbwh78Grv2KEEKoZ7BH2E2v73zH6XOF7XBRtz3hrTjWsOYNwEnSEUIoqDARdtPlgy+WsuGbJ5BI5dH3PuspK67/6EUQwjT/HkII9QeYCLspSmEoaTgRziMyclXMvc8JNkv9h8so7+t4A4QQQp2AibCbTtlryqwVYT4okcqi7lgiiTeF+bgIIdSH4WCZbhpqHDzUOLi6pjrMxyWcVHvZH8N8UIQQ6sOwR9h9z/7y8kl7lZgRUEo94RuwgxBCfRImwu57IvOhJFWCiAE49/xc89pjgj0cKzgjhFBfhYmw+yptp1aXrBcxAMW4GfKMydWvPspb6kQMAyGEejVMhN1nkOumxk0QNwbt7JtVky+uefVRX52oJ2kRQqjXwkTYfXKJnCFMiSWsN1G0pLnoevXMa2tfX+CrCfcoVoQQaofZbM7KyjIajdnZ2WazucPa6urqm2++OSEhITk5+a677mpsbAxPnJgIe6TKUe3hPWJHAerpV2kuudF9ZK/YgSCE0FlLly41mUyVlZWpqanLli3rsPbPf/7zwIEDS0tLi4uLDQbDk08+GZ44Ce3ulF08z3u93uBG0+vU1tYqNMpGny1eFSt2LOKz2+08z2u1WrEDiRRVVVXx8fFiRxEpLBYLx3FKpVLsQCKC1+u1Wq1RUVFiB9Jlu3fvPnjwYEBhQUHB/v37N27cGFA+bNiwtWvXpqenFxUV5eTkHDp0qP1ajUZTXl6u0+kAwGw2jxs37vjx46F8Nad1/z5CQRCcTmcQQ+mNnE7nPmuRQARdvEbsWJqhVJS5uV0uF8/zHBe+mecinMvlwr+RJv6PB8FZ4wEAwOv1Op3O3vjxKCws/OGHHwIKKysrW/2XraioMJlMAODv+XVYO3HixBdeeGHBggUej+fZZ59tuUmIdD8RchxnMBiCGEpv5PP5LjEMlEgkNo9dLY2MtQMprfnX49pLbpSljQ3zkaVSKfYIm3O5XPg30oRhGOwRNvF6vQzD9MaPx5133nnnnXcGFK5fv37lypUtG1NK/QmSUsrzgZMkt6x977337rvvvpSUlNjY2Pnz5xuNxpC8hhbwGmEQ7D61b83hdWJHcQYh2tm31L3/vHPfL2KHghDq1xITE8vKygCgoqIiKSmpw1qVSrVmzZrGxsajR4+OHTt22LBh4YkTE2EQjIsbdcvI64WIWSBJNigj+u5nGj5b6dzzs9ixIIT6r6ysrLy8PEppXl5eTk6OvzA/P7+t2scee+zuu++2Wq2VlZULFy6cP39+eOLERBgcxxpK/7m9lTMDYpGmpEXf90LDl2/at38ndiwIoX4qNze3oKAgJSWlsLBw0aJF/sKZM2e2Vfviiy/W19enpKRcdNFFf/7zn6+55prwxImTbgfHQH3q386bJ3YU5+DiU2Pue74u7ynF6GmMQi12OAihfkev169bF3jZqOlWhZa1UVFRa9euDVNwzWCPMDgIkDpH/bO/LBc7kHNIYpNjH30dsyBCCLUDE2HQRCujHpx0j9hRBCKSszczNG7+vGH1v3BiUoQQag4TYTARIMt/+xeFSBk1E0A1+RJGqTm17N76j1701YXpBh2EEIpwmAiDSckprkvPIhChtwwzap129s3xj7/Jao3VL803f/Ya9g4RQggTYZAlaRJWHfhCoILYgbSJUet1WXfEP/EOo1ALTpvY4SCEkMhw1GiQsYQdpDeJHUXHGJVWd9XtZ58LvM9cI4nCiTERQv0O9giD77zEiT+X/crTwPmEIpm3srR6+Xzzxy/7avHaIUKof8FEGBJmV4OX701Lc3BJgxJy35fEJlW//EDde8/6qsvFjgghhMIEE2FIXD30ikp7tU/oTZ1CIlNoLro+fvG7XMKA6lcfadz4idgRIYRQOGAiDJXNx3+2uK1iR9FljFylvWxu/KI8xZjpYseCEELhgIkwVO4YM1chkXsFn9iBdAcjV0lizk4V7yk5IGIwCCEUUpgIQ+jdglXljSfFjqKnBIfN/Mkrtf/6u7eqVOxYEEIo+DARhtD9E+4YqEuN5HsKO4NRquMe+5dizPTa/3vc/Omrgs0idkQIIdQ1U6ZM+fjjj9uqxUQYWm/t/XBvdaHYUfQYw6oyr4hb8AZh2KoX7nbs2Ch2QAgh1AWxsbGbN29uqxYTYWjdOebmcXGjxI4iOBiVVn/d/THzljJao9ixIIR6AbPZnJWVZTQas7OzzWZzZ2q3bNkyduxYjUYzduzYn376KViR5Obm7tmz57HHHvv555+LmvHXYiIMLQLkqyMbNh0P2j+n6Lh4k3zY+KangqNRxGAQQpFs6dKlJpOpsrIyNTV12bJlnam9+eabn3jiifr6+r///e8333xzsCKZPHny77///uKLL86YMWN4M/7a7k+xRikVhN599avneJ7neZ6Q9mbZnpV6vkIi5/nedE9hJwlOW+3SexSTLlJddAMjV/JniB1XpMB3ozme5xmGwTfEr/f+sdTW1rbs25WXl7eaDtasWbN27VqZTDZv3rycnJznn3++w1qtVmuxWGw2W2Njo1odtLVUm1YDbhVpv7odbrfbYunv4ybq6up0Op1E0sHviT21+83uhplJffHOPKeN3/olX/irZFq2a3gmT6lGoxE7pkhx6tSpuLg4saOIFFarleM4hUIhdiARwev1NjY2Go297yrDq6++umrVqoBCh8MxZMiQrVu3BpSr1eqamhqFQuF0OuPi4qxWa4e1v//++6RJk/wNduzYMXHixJC9lLO6nwgRANTU1BgMhg4TYb3TrOAUCok8PFGFn6fsSMPqNwSPS3bZLYbR54kdTqSorKxMSEgQO4pIYbFYOI5TKpViBxIRvF6vxWKJjo4WO5DgWL9+/cqVK9evXx9QrlKp6urq5HK5w+GIiYmx2+0d1l500UUTJ0588MEHX3755V27dm3cGLSheV9//fWyZcsOHjwoCEJGRsbChQuvvPJKfxVeIwwHo8JQZTvVl64UBpCmpMXO/6dseraA6/0ihM5ITEwsKysDgIqKiqSkpM7Ubt++/aGHHkpISFiwYMH27duDFclnn312zTXXnH/++V9++eXXX399/vnn5+TkrF692l+LiTBMlJwizThI7ChCiRDp6OnchIuaCixfvln3zj8cv/2AA2oQ6p+ysrLy8vIopXl5eTk5Of7C/Pz8dmpHjx79zjvv2Gy2Dz74YMyYMcGK5Lnnnnvssceee+656dOnT5s27bnnnnv00UefffZZfy0mwjCJU8VqpJrfKneJHUj4aC77o2LM+c79v1Y99aea/3vctu0b3lovdlAIofDJzc0tKChISUkpLCxctGiRv3DmzJnt1Obl5a1fvz4hIeHzzz9/++23gxXJ4cOHp08/Z5TGBRdccOjQIf9jXJg3fJw+p693Tj3aPYxCrZw4SzlxFvW4XUU7nQVbwetRX3iN2HEhhMJEr9evW7cuoLBpYEqrtenp6du2bQt6JCaTqbCw8PLLL28q2b9/v8l0ehF1TIThk6iOj1PFHDEfSzP06XOkLRCpTDE6UzE6s3mh9buPAEAxehqXMECcsBBC/cZdd921ePHiuLg4/wCZdevWPfXUU08//bS/FhNhWNU66nZU7u5vibBV8vQJjt0/1b21BFiJbPBIYFh5+gTF6Gn+WlfRTmfBOb8KCSfTz7nb/5h63A1f/ptRqBWjMqWmYdDufZwIITR//nyfz/fQQw/deuutABAVFZWbmzt//nx/LSbCsIpTxf5xxLX1TrNRYRA7FpFJTelSUzpcfZe3vNhTdgQoZXVRTbWsxiBNHtK8PZFwZ58wjDR5CG+pM/93ueBxKcfOUIybIU0ZGrbgEUK9C8Mwjz766COPPFJTUwMAMTExzSdCwUQYbha39Y097/996oNiBxIpuOQh3Lk5DwC4pEFcUpv9ZiLhVJlXAID28lu8lcedu7eYP/pnzLwXGbUutLEihHqnKVOmPPjggzfddFNsbGzLWkyE4aaTaf8+9cESy4lK26nMpElih9PrcQkDuIQB2iv+1FQiOGy2LWsU42Zw8aYwB0N9Xl9VqeC0CU674LTT+jqfZEbzJY4RQqLwrz5x0003tVqLiVA0iep4p8/l8rkNcuzHBJXgox5X7b8XMzKFYtwFynEzJLHJ3dsT5X1Co5lwMkal9Zc49/3PXbxXcNqp0+ZPeKwuOvqup/y1vrrK+v++zCjUjELFyFXU6axZ+U3M/JckUfHBeWkIoW7Jzc297777HnvssaysrJiYmKby9PR0wCnWeqiTU6y1ZWv5dofXeenAC4MalGjsdjvP81qtVuxAAACAUk9pkWP3T849P7G66NiHXml1TA3lfUJjA6uLaqo1f/Yab67mG2qFxgbB0cho9Lqr/qycOMtf6z6821t5nPhTnf//Ki2rj2m5Z/BPsRYf37Rn6vNavnpbMfI86eBRhO13v0FxirXm+skUa5GjraUR/Bmw3/01RpTpyVMAYHPpz27ec/mgizpsj7qAEOmA4dIBw/VX3+U9VdaUjbynTtg2f8E3mnlLrWA1C45GRq2L+9v/NV1flA0ZzciUrD6a0ehZTeCYJtnQcbKh47oUxtnHvI/VRVnWv++rrpCPmKQYNVU+fBKR9tkZaBGKHO13+TARii8zaTIALWkotXkdo2KGix1On0MIF5/a9IyRyqUDR7BaA6uLYjRGVqMP6Ckqx10QqkBkCs1F12suup631rv2/2r/3wbzf1fE577HKIK21gxCqFVNg2VarcVEKD65RAYANq9DI1U7fS4CIO+761SIjjXEqs67TOQYtEZV5hWqzCuox9XUI6Rup+1/3ypGTZVERd6CFYLAN5r5hhreUs831PDWekap0cy8Fu/gRL0FDpbpHfx9wW+PbVJI5BemThM7HBQOzc+LCm6Xr7q8ZsXDjErHGuNYlZZR6zSX3NjUX+QtdYxcSWShWs+Petz+JMeba3hLrSQmUTHmfH+Vfft31g3/YXXRrC6KNcQwGj31uJqyoOCyW758i4tPlSQM4OJNze8HRShCtD9YBhNhZPFfKfy+5EcVp5qWPFnscFD4sFqD4fr58Ie/esqLeUudYLcIjQ2EYZsa1L39D++pUqCUUekYtY7V6FVTLlOMPZ2rfKfKvJXH/Y8Fjwt8XiJXKsdfeLrEabNt+ZJ6PU1PAUBzwRxJXIq/pPbfi93Fe1l9DKszsvoYVhdFuLNJWjX1ctXUs5M0BiCEkaYO9VaVOg/85qsspbyXizcZbniwaecIiW7y5MkA8Pvvv7/44ovNy3GwTOSalDBeIZGVNJQCwEB9uG+GQ2IiRJqSBilpLWtiH3kVAKjHLdgtvK1BaLRIos+eRHUfP+g6uMP/mJHKQcI13fIBAEApADDK051L/+0czTuXUX9eTDhpN0OWKfzzG/gJdqu3spTVnx0PWb3iYSLhqCGON8bxMjkjU3DJQ6Spp2cC4q31gtVMFErCckQqJ1LZObMIdYLgtIF/JITgE9xOIpW3HOWE+jkcLNP7+O8srLRXJ6njXT43x0pYwna4FeoPiFTGSmNZQ+DsGKopl6qmXNrWVoxSo519c3u77W4WbOVYKq1syKjmJVG3L/ZVlVpLigS7RXA7fG5n89Onrv2/2n5ZR50O6vNQr5u6XYxal/CPj/y1gs1Ss/Jv1Os9/dRpA6DGWx+Xp0/wl9S+mespOQD+07SMhJEpZMPGGa4/PYek68Bv5k9fZVQ6VqNnVDpGrWVUOs2F1xCprGmHOFgJYSKMXP55Zz49+GWKNnlq0kSxw0Gom1itgdUapHEDW72P0D9uqK1tiUJlvG0R4U73ERmFGoAQ2dnTtk2zGbRKNmx87IMvCzYL39gg2K2C3SLYrMCe/Vl56oW7+UYzq9IxKi2j1jEqnWrKpfIRp6d8cuze4i4uYOSq08FIZawuummwleCwOff8RDgZ4aRACKNQAYDUlN7U1RZcDhCEbvRx+wyz2Xzrrbdu27Zt+vTp77//vsFg6LC25Q1/QbnZnVL6xhtvrFq16siRI/n5+e+///60adOuuuoqfy0mwkj3h+E5BMi3xzYlqOPGxo4UOxyEwoqwkuZ3v3Rjc1Yfw+pj2kpECf/4iPI+wW4V7FbBZhHsVkns2SnxJPpoIXEgdTn8T6nHRd2OplrqdXvKi6nXTb0eoFRw2gFAP+fuppXF6t971nPiEPW4qc9LJByRypTjZ+qvu99fKxw/0JC/zz+tLhdvCtEdpdTn9VWX++qqJIaYlpP6htrSpUtNJtPnn3/+yCOPLFu27Pnnn++wtrGxsanBsmXLPB5PUCJ5//33//rXvz7wwANbt24FAKvVmpWVtXr16jlz5gDOLNNDPZxZpvNO2qqMckO1o1YjVUfslGyRNbNMBKisrExIiLx7IUTSz2eWoV4P9Xr86RAAvF5vw/EjipOHvCdLPBXHfNVlrC5aahpmvPmxoBzO/ut3zr1bfdXlvLVOEpXARsVLk4doL7/FX+s6sMP82WuMSsOq9YxSw6i0jEqrmXVdUzLmG2oZharzQ5Tbmllm2LBha9euTU9PLyoqysnJaVoRvjO1+/bte+ihhzZs2BCUL9gxY8ZceeWVzz33HCHk4MGDw4YN+9Of/lRUVPTbb79BT3qEbre7oaGh5/H1anV1dR6PJwyJkAVisTdsKd860jjco4wL9eG6x+Fw8DzvdDrFDiRS1NTUMAwjdhSRwmq1chynUITq9o/exev1NvqIMT0T0jMBQCLwtP6Ux1p76tSp0y0sdb6v3iCxKSQ2hcSlkphk4GTn7EIQqKUG6ippXSWtO0nrqpjxM5mM08tfU0YGI6dDVIJEFwMMwwM4AZxNOzckMTctpE6r12EDpw2cNrDbnbV1cGbmP99bj1NLHTNmBntJ4KXlZcuW5eXlBRT6fL7Ro0e3fJkVFRX+VeBNJlNlZWXnaz0ez5133vnuu+8G69v18OHDy5Yta3pKCJkzZ84tt5z+ZdD9Y8hksuZ3Y/Rb4ekR+s2N+QMAvLrzzewhlw/QRdzYdOwRBvD5fPg30kQqlfbnHmEAr9crlUrPmWs07pyZ2ale582+3XuyxHuyxFe4zXvqBKuLNt79jMR4+nfwqaduJSzHxiRx8amSYWPZmCQuaVDT5Uzo8IMX1+5E8H9/u62af/zjHwsXLgwo3LRp0/vvv9+yMaXUf82PUsrzfOdrX3rppcmTJ48YMaK9ILvCZDL5VyJscvz48dTU02fde/QNjr92mTPCedC/jL1VxSl/Pfn7iKhhWpkmnIduH8MwlFL8VDQJ/2cjkonyxxKxOn43ZHJ26Dh508S2Au+trpAYYsiZTeIXvyfK1O1qtVqtDhxnq9VqW53VOjExsaysLC0traKiIikpcD2ytmp5sM4TEgAAIABJREFUnn/jjTc2bdoUxLAfeOCBF154YerUqQDgdDq/+eabJUuWrFixwl+LH8reR8UpAaDGUSeTyDx8cK4kI4QiGsNy8anNM1+vWMAkKysrLy+PUpqXl5eTk+MvzM/Pb6cWADZv3pySkjJkSDCH9txzzz0LFiy4/vrrVSrVZZdd9tJLL/3nP/+5/fbb/bWYCHurrCGXyVjpkp+X2Tx2sWNBCKFW5ObmFhQUpKSkFBYWLlq0yF84c+bMdmoB4L333rv00jZviu0eQsgtt9yyc+dOm81WXV39448/ZmdnN9X2gt8UqB3PXvB3hjAfH1h97bCrpGzQ7olGCKGe0+v169atCyhsulWh1VoA+Oijj0Ie2bmwR9i7MYShQE26ZCkrrXeaxQ4HIYR6H0yEvR4Bkpk02Sv4Xt7xhoB3hSKEUBdhIuwjOEby9IzHPbx7xe//FjsWhBDqTTAR9ilSVpY1ZDYAHDEfEzsWhBDqHTAR9ikMIYP0pnqn+acT/xM7FoQQ6h0wEfZBRoXhjjFzyxtPrvj9TbFjQQihSIeJsM9K0iT8adQNTp/rpzLsHSKEUJswEfZZBIhepqtz1ktZDgDe3vsfsSNCCKFIhImwj0vWJJ6XOFGggn8tw//blWd29fc1QxBCqDlMhP0CQ5iJCWMB4MrBlxjk+k+L1m4r/02g1OVzix0aQgiJDBNh/2LSpQDAnKFXTk4Yd8R89J2CjwDghLVC7LgQQkg0mAj7I46RcCw3zDjkvvG3e3nvqsLPAWBfzQGXzyV2aAghFG6YCPs1AoRjuYVTHwCA3yv3soTdV3PwuKVM7LgQQih8MBGi024ffRPHci6fS8ZKyxtPfnt0IwDUOuvFjgshhEILEyE6x6SEcQnqOLlEPiwqDQD+uX0lAOSf2HbCWsFT3o3rACOEOs1sNmdlZRmNxuzsbLM5cHmcVmt9Pt99990XExMzbdq0ioowDV/ARIhaEa0wDtKbAOCFC3MBwCjX62Sa4vqSvL0fAcDnRV8DgE/gxQ0SIRThli5dajKZKisrU1NTly1b1pnaV155xWq1lpaWZmZmLlmyJDxxYiJEHRsdm6GTaYdFDbl3/O0AoJWpAeDdgo/2VO9vcFu2lm8XO0CEUCRas2bNvHnzZDLZvHnzVq9e3ZnaVatW/e1vf1Mqlbm5uffee2944uz+CvUej8dmswUxlN7IbDZTSiWS7r+NvdFE3Zj6+vprU68SKD1VW+2wOerr6//+63NPjH6guKEkXh8Xp4wRO8aI0NDQIJPJxI4iUlitVo7jXC4cmQwA4PV6rVYrw/S+rsh//vOflsvK19TUaDSalo0rKipMJhMA+Ht+naktLS39+OOPZ86cOWjQoHfffTf4L6A13f8Gl0gkKpUqiKH0Rk6nU6VS9bdE2JxGrR4QnQoAL164xOvyOixOqVxay5t/KM2/c9TN+2sPZkSnEyBihykOpVKJfyNNfD4fx3FKpVLsQCKC1+vleb43fjwuuOCClJSUgMJdu3b9+uuvLRtTSgkh/gc8H3gxpdVaq9VKKS0sLHz99df/8pe/tLrboOv+NzjDMPhrVyqVymSy/pwIm8hAZqf281OmarVan8DP1V4nk8l+rdo5IWnsuqM/DNSlphkH1zrqEtRxYkcaPv6Ph9hRRAqZTMZxHL4hfgzD9NKPx5gxY8aMGRNQKJPJdu7c2bJxYmJiWVlZWlpaRUVFUlJSZ2pjYmIefPDBhISEefPmvfLKK6F4CS31vo45inwShtXLdABw/4Q7AGBc3KhUbXKl7dR3JZsBIK/gI4FSp88lUEHkQBFCoZSVlZWXl0cpzcvLy8nJ8Rfm5+e3U3vZZZe99957brf7zTffnDhxYnjixESIQi5RHa+WqlK1SbeNugkAMqLTGUJWH/pmZ9Veq7tx4/EtAMBTHIOKUF+Tm5tbUFCQkpJSWFi4aNEif+HMmTPbqX3++ec3b94cFxe3adOmt99+Ozxx4jk9FG5TEicAwNyM6wCgzmnWSjUA8MAPT7x6yfMFNYVaqWagPtUn8ByDH06Eeje9Xt9yZA2ltJ3a+Pj4H374IRzBNYM9QiSmKIVhcuJ4AHj1kucZQhggconslL1m2a+vAsCXh9d7ea+b9+BJVIRQ6GAiRBGBIQQARsdmJKrj41WxT2Q+DABqqYpjue+Obd5c+rOH93xW9BUANHr6+007CKHgwkSIItfFAy4AgOy02RcPuMAn8CZtMgD8c/vrdq9jf03RnlP7AKCwtkjkKBFCvRwmQtQ7KDmF/yTqP85foOKUGqlKL9f7BD6/dBsALP311eOWshPWiu+ObQaA3yv3AE4ChxDqHEyEqFcy6VIG6FIkDOu/Q+ORyfenapPVnHKwYSAA7DpVAAD/Pbhma/l2q6fxvX0fA8BvJ3e5eY/V3Wh2WQBgZ9VeANhfU1TrqDO7LD+WbgWAD/Z/4hV8v1fu+ansfwCw6KfnAOCtPR8U1hadtFXlFawCgE+L1voE/qStqrzxJOACHQj1fpgIUV8gYViGEKPCMMQwEADuGnsrANyccd305ClShpuePAUAjllKGSCH6ot/O7kTzvQa65z1bt4jYVgpKwWA4VFDCZDBhgHDjIMBYP7EuwDg1lE3pkelRSuMVw6+BABildEShj1lrzllrxGo8MqOfwPAyp1v+7uk3xR/BwDfl+QDgM1nd+JaxwhFPNI0khV1Q01NjcFgwJll/Ox2O8/zWq1W7EBEYPc6ZKzM4XPUOOoG6wd8XvT1delZq3Z9PjBuwJjYjE2lP2UNuYwC7bezzQGAxWLBKdaaeL1ei8USHR0tdiDBsX79+pUrV65fv17sQLoJe4QIBYGKU0oYVivVDNYPAIDr0rMAYGbCtKlJE32CL14VCwAPbVzkFXwllhOn7NXiRosQag4TIUKhpZVpJiWMA4BlM5dwjKTMWtHosVU7av1XHE/Zqyn09KyMf8o6AHD6XAKlPgGXUEaoCzARIhQm/suQM1KmDjEM0ko1swfNAoD/2/UupbD95M6ShlIKtGVSpECtnkYAOFx/1OVzn7RV7T61DwD+tetdAFj9/+3dfXQTZd438Ctt05f0LQQK2NrWmwoU6Mrycrv3Yi1UFIQ1DaB7nj2ASB9c15U+x7UccVdZ2EVU2lVWHkBFjhFEpShLLVhEBFqgwGKhQLG0pVBq25C+JWnSZNJkMrnuP2bNZtPSN5pMynw/f6XXbyb5ZbiSL8nMZKoLv9eWmeym3H9uIYS8fW6bvtNwsbmcP+fyucPZhJBdV/aWNJ4zdBr/WvI3QsiG05t0VsPF5iv8Mh+X7/HhNgDwSxTuQEtLC8uyQnfhL8xms9FoFLoLP3Lr1q0+LlneUtFqabtuuPnOufcopZtLP6SUHqz5tujHEitrXXPiTUrpZxX7WhmdpkN7TnOBUnpBe5lSarZbrKy1Lw/BOZ0Ma6WUMqyVczqtbKfJ1sE5ubOaUv4Rm8zNNoeNc3IDeap90N7ebrFYvHTnQ47dbm9tbRW6i0FTWFg4b948obsYOBwsc0dwsIw7MR8s0y2tVnvPPff0axXW6ZAGBJW3VDwwcpKFZYIDpNJAqZfac9fCtMXIhn93s5h1OuaNebTGcGO84v7BfQgcLOMOB8v4FXw1CuBH+J8af2DkJEJIuFTmmxQkhIyUjZAQyZz/Sp+f9KjOqiu9dZEQ8s2No078RxlEAEEIAP8mIZIY2YilKb8mhNidbIBE8tW1Q3XGBi89HH/5LUOnsc2qd1LnsR9PEUIO1BxutrReN9Ty+y+f+yabEPJt7fET9WccTu7/n99BCPn6+rdtVn2zpeWft84TQk41nCWEtDBtOquBEsr/yoGh0+hwcgxrbWXaCCG17T8SQpotLXqrweHkLrX8QAi50nrVwjJtVv3VtmpCyIn6M156puDPEIQA0D3V2HmEkJSY5HsiRpU0nvuiqoAQorMael7LSZ1mu4UQUm/SOClts+o1HVpCyNmm84SQY3Un+R/l4UPuxe9edTi5q23VVbqaAEmAjtERQsYpksKCQhOi7l04fj4h5MN5mwghqfH/82Ds1ABJwNwx6YSQhKh7w4JCCZEESgIJIT8aGwkhV9uq602NNoftvTI1IeSzii9bmNaGDk1R/WlCCN9/pa6mztjgpNzFpiuEkNr2H+2c3cpaW9zC8tOKfReaLjOslc9XwS9+0sboCCEtTFu7zUgIMdpMwvZzFxJ6J+XQhoNl3OFgGQ99P1jG/3FOp5W1dtjMG06/Qyn9uPzzToet9NbFb2uLKKV/Kn6dUrrtwkdlTeWaDi1/yM/b57ZZWevFpvLTjecopTtKP7FYLBWt1a2Mzu6wa83NQj6fHtkcNrvDrrcajtadpJT+vyN/tDvsV1oqK9uuUUo7bOaB3a2TOimlmg6t3WHXGptO15yjlO6rOkgpPdNY+kNrJcNat5zfQSl9t/QDrbn5vPbS+2Uf058275GbRRebr+ithg2nN1FKv7p2yMpaWc4xGM+4e30/cup2B8vo9fonnnhi2LBhSqVSr9f3pTpjxgxXPP3ud7+7k/77DkF4RxCE7hCEHu6mIPRwufkHu8NutJl0jJ5S2sroel1l6B41yudBZdu1H40Nxk7T2pMbKaVfVBZ02Mxmu8VoM1FK7Q47pVTToWVYq47RX2y+QindWb6HUnq07mRJwzmbw7by21copR9d/lTToa3T1xdcOUQpPXTjKKW0rr2+ydzMObkafS2lVG9t7zXhjtaddFLngZrD39w45nA6Dt84dufPtLb9R6PN1GRu3ld1gFJ6rO5kH1e8XRC+8sorK1eu7OzsXLly5R//+Mdeq06nU6FQNDY2dnR0dHR0WK19OiL6zg38qFGHw9HZKfbfUWxra5PL5ThqlMcwDMdxkZGRQjfiL5qamkaPHi10F/7CZDIFBQXdNUeNnm++NHXk5PMtF812y6x7U18p+evfHv7LvusH/2f09JDA4Kv6azPjfnmuqezB0VM67GYJkUQGR7ivzrKsyWQaPnz4nXfipNTqsH7fXJZ+b+pf//m3l6dnGTrbOepMiIzrYS1DZ3tkcGQz09LMtE4d+cBrZ958Y8arxxtOJUTeGx8Z96OpYdywpG5XLCoqKi0t9Risqalpbm4+fvy4x/j48eMLCgqSk5OrqqpUKlV1dXXP1aamprFjx44fP766unr27NkffvjhyJEj+79J+u2OgtBqtQ5uN0MOgtAdgtADgtCdyWSSSqVhYWFCN+IXBjEI3dk4W0hgyOXWCmmg9L6o+PfLd66a+vtr7TcSI+N1nXonpaNlI7f/sGvlA//346t7Hk98JCggSGtpfmDERMZhlQX16Z/m6NGjZ8+e9Risra3V6XRFRUUe4xEREa2trWFhYVarddSoUSaTqefqpUuXsrOzN23alJCQ8NJLL9nt9j17fPGDDziP8I7gPEJ3OI/QwwDOI7yL4TxCdz44j9BJnfrO9hFhih2Xdy9L+T81+lqrw/rf90y5bqi9f9iYwX2s251HGB4ertPpQkNDGYaJiYmxWCx9r2q12kmTJun1vrjMGY4aBQC4CwVIAkaEKQghv538dEhgcEpMMv+bt4Oegj2IjY1taGgghGg0mrg4z69qu1bLysrOnPnXGSzBwcEhISG+6RNBCAAAXqFUKtVqNaVUrVarVCp+sLi4+HZVi8WycOHCyspKu93++uuvL1iwwDd9IggBAMAr1q5dW15eHh8fX1FRsWbNGn4wPT39dtXU1NR169Yplcq4uDiDwZCTk+ObPrFzCwAAvEIulxcWFnoMug5M6VqVSCQvvPDCCy+84KP+foJPhAAAIGoIQgAAEDUEIQAAiBqCEAAARA1BCAAAooYgBAAAUUMQAgCAqCEIAQBA1BCEAAAgaghCAAAQNQQhAACIGoIQAABEDUEIAACihiAEAABRQxACAIBXGAwGpVKpUCgyMjIMBkPfqz/88EN4eLjP+kQQAgCAV+Tk5CQmJmq12oSEhNzc3D5WjUbj8uXLGYbxWZ8S1zUS+8vpdLIsO7jdDDltbW1yuTwoCNc3JoQQi8XCcVxUVJTQjfiLpqam0aNHC92FvzAajVKpVCaTCd2IX2BZ1mQyDR8+XOhG+u3GjRv19fUeg6WlpadOnTpy5IjH+Pjx4wsKCpKTk6uqqlQqVXV1da9VSumiRYuWLFny61//esDx1F8Dfwd3OBwWi2UQWxmKLBaLVCpFEPIYhuE4LjAwUOhG/AXDMHiNuDAMI5VKffbW5udYlrVYLKGhoUI30m/ffPNNQUGBx6Ber1coFF0X1mg0iYmJhBD+k19fqjk5OUlJSU899dTgt357A38HDw4O7vaZiwrHccOGDUMQ8kJCQvCJ0J3NZsNrxCUwMBCfCF1Ylg0MDByK02PVqlWrVq3yGDx06NDWrVu7LkwplUgk/A2O43qtFhUVHT58+LvvvvNK67eHfYQAAOAVsbGxDQ0NhBCNRhMXF9dr9dixYydOnAgODuYDUiKRlJSU+KBPBCEAAHiFUqlUq9WUUrVarVKp+MHi4uLbVTds2EB/QgihlKampvqgTwQhAAB4xdq1a8vLy+Pj4ysqKtasWcMPpqen91AVBHZuAQCAV8jl8sLCQo9B1wFT3Va7LuYD+EQIAACihiAEAABRQxACAICoIQgBAEDUEIQAACBqCEIAABA1BCEAAIgaghAAAEQNQQgAAKKGIAQAAFFDEAIAgKghCAEAQNQQhAAAIGoIQgAAEDUEIQAAeIXBYFAqlQqFIiMjw2Aw9KV6+PDhiRMnyuXyiRMnHjlyxDd9IggBAMArcnJyEhMTtVptQkJCbm5ur1Wn07lkyZItW7bo9fr169dnZmb6ps87ujCv0+kcrD6GKOdPhG7EL2BreMDWcIfp4W7obg273c6yrMeg1Wrt9jq6+fn5BQUFISEhWVlZKpXqrbfe6rnqcDh27979yCOPmM3mkJAQuVzuxWfiZuBBaLPZ2tvbB7GVoUin0zkcjqCgO/r/xF2DYRiO42w2m9CN+AudToe54WIymaRSaVhYmNCN+AWWZTs6Onx5EfbB8uabb+7YscNjkOO46dOnd11Yo9EkJiYSQvhPfr1Wg4OD58+fbzabo6KiJBJJSUmJV55DF5Kh+C/hP1pbW4cNG4Y3O57FYuE4LioqSuhG/IVWq73nnnuE7sJfGI1GqVQqk8mEbsQvsCxrNBpHjBghdCOD49ChQ1u3bj106JDHeHh4uE6nCw0NZRgmJibGYrH0sWqxWDZv3pyfn19aWuqD/rGPEAAAvCI2NrahoYEQotFo4uLieq3W1dW9/PLLhJDw8PAVK1ZUVlb6pk8EIQAAeIVSqVSr1ZRStVqtUqn4weLi4ttVY2NjP/rooxMnTlBK9+7dO2XKFN/0iSAEAACvWLt2bXl5eXx8fEVFxZo1a/jB9PT021WDg4Pz8/Ozs7OHDx+el5fXdWekl2DnFgAAeIVcLi8sLPQYdB2Y0m115syZFy5c8EVzbvCJEAAARA1BCAAAooYgBAAAUUMQAgCAqCEIAQBA1BCEAAAgaghCAAAQNQQhAACIGoIQAABEDUEIAACihiAEAABRQxACAICoIQgBAEDUEIQAACBqCEIAAPAKg8GgVCoVCkVGRobBYOhLtaCgICUlRS6Xp6WlXbt2zTd9IggBAMArcnJyEhMTtVptQkJCbm5ur9X6+vqlS5fu2LFDq9VmZGRkZmb6pk8EIQAAeEV+fn5WVlZISEhWVtb+/ft7rdbW1v7mN7/55S9/GRYW9swzz1RXV/umT4nrYsH9ZbfbTSbT4HYz5Oh0uujo6KCgIKEb8QsMw3AcFxkZKXQj/qK5uXnUqFFCd+EvTCaTVCoNCwsTuhG/wLJsR0eHQqEQupF+e++99/Ly8jwGzWZzYmLiqVOnPMYjIiJaW1vDwsKsVuuoUaM8IqOHKsdxWVlZAQEB27Zt89ITcTfwd/CgoKDo6OhBbGUostvtCEKXoKAgjuOioqKEbsRfMAyD14g7qVQqk8mE7sIvsCxLCBmK02P58uVPPPGEx2BJSUlhYWHXhSmlEomEv8FxXB+rR48eXb169Zw5czZs2DDI3d/GwN/BAwICAgLE/s2qVCqVSqUIQp5UKg0ICJBKpUI34i/46SF0F/5C+hOhG/EXQ3RrJCQkJCQkeAw2NTUFBgZ2XTg2NrahoWHs2LEajSYuLq7XKqX01VdfPX36dF5e3rhx47z0FLoSe5IBAICXKJVKtVpNKVWr1SqVih8sLi6+XfXMmTP5+fkHDhyIjY01m81ms9k3fSIIAQDAK9auXVteXh4fH19RUbFmzRp+MD09/XbV4uLi6urqYcOGRf7EN33iOz0AAPAKuVzedd+h6wjNrtXXXnvttdde81FzbvCJEAAARA1BCAAAooYgBAAAUUMQAgCAqCEIAQBA1BCEAAAgaghCAAAQNQQhAACIGoIQAABEDUEIAACihiAEAABRQxACAICoIQgBAEDUEIQAACBqCEIAAPAKg8GgVCoVCkVGRobBYOhjleO45ORkX/aJIAQAAK/IyclJTEzUarUJCQm5ubl9qW7evHnGjBnV1dW+7BNBCAAAXpGfn5+VlRUSEpKVlbV///6+VB944IE///nPPu5z4FeoZ1mWYZhBbGUoMplMAQEBQUED34x3E4ZhOI5zXX4aTCaTTCYTugt/YTKZpFIpy7JCN+IXWJblN4jQjfRbQUHB8ePHPQYbGxsdDkfXhTUaTWJiIiGE/+TXl2p6evrgN92bgb+DBwQEDMV/xcEllUqlUimCkBcUFIRZ4Y6fHkJ34S/4Vwo2iMsQnR4JCQnTp0/3GAwLC7t27VrXhSmlEomEv8FxXL+qvjTwd/DAwED8bzcsLEwmkyEIefxsxqxw4aeH0F34C5ZlpVIpNgiPZVmWZYfi1pg1a9asWbM8Bg8dOrR169auC8fGxjY0NIwdO1aj0cTFxfWr6kvYRwgAAF6hVCrVajWlVK1Wq1QqfrC4uLiHqiAQhAAA4BVr164tLy+Pj4+vqKhYs2YNP+jaC9htVRD4Tg8AALxCLpcXFhZ6DLqOp+u26rGMb+ATIQAAiBqCEAAARA1BCAAAooYgBAAAUUMQAgCAqCEIAQBA1BCEAAAgaghCAAAQNQQhAACIGoIQAABEDUEIAACihiAEAABRQxDekba2tm6vyyxOFovFZDIJ3YUfaW5uFroFP2I0GhmGEboLf+FwONra2oTuAv4FQXhHFixYUFNTI3QX/uLTTz9dv3690F34C6fTOXXqVKG78CPr1q3bs2eP0F34i6tXrz755JNCdwH/giAEAABRQxACAIBXGAwGpVKpUCgyMjIMBkNfqj2v4iUIQgAA8IqcnJzExEStVpuQkJCbm9uXas+reMnAr1B/8+bNI0eODGIrQ1F7e/sXX3xx5swZoRvxC2fPnm1qatq+fbvQjfgF/hLb2Bou1dXVVqs1ODhY6Eb8Qn19vcFgGIrTo66u7tatWx6DGo2GZdmuC+fn5xcUFISEhGRlZalUqrfeeqvXas+reImEf7kOwIULF7Zs2TK43Qw5DMOEhIQEBgYK3YhfsNvtlNKQkBChG/EXZrM5IiJC6C78RWdnZ2BgoFQqFboRv8BxnM1mk8lkQjfSb3V1dRqNxmOQUvrOO++oVCqP8YiIiNbW1rCwMKvVOmrUKI+jyrut9ryKlwz8E+G0adN27tw5eJ0AAMBdhVIqkUj4GxzH9aXa8ypegn2EAADgFbGxsQ0NDYQQjUYTFxfXl2rPq3gJghAAALxCqVSq1WpKqVqtdn1xWlxc3EO120FvG/g+QgAAgB60t7cvWbLk8uXLU6dO3b17d3R0NCFEIvlX7nRb7XbQ2xCEAAAgavhqdOAeeughyU+ef/55odsRDMdxycnJ7iOCnBLrJ7puDdHOk4KCgpSUFLlcnpaWdu3aNX5QzHOj2w0i2unhVxCEA0Qpraqqamxs7Ojo6OjoePfdd4XuSBibN2+eMWNGdXW1+6Agp8T6g65bQ7TzpL6+funSpTt27NBqtRkZGZmZmfy4aOdGtxtEtNPD71AYEK1WGxERMW3atIiICJVK1dzcLHRHwjh+/PjBgwc9JtK4ceMqKysppZWVlePGjROoNQF03RqinSdFRUXPPvssf7ulpWX48OH8bdHOjW43iGinh7/BPsIBunTpUnZ29qZNmxISEl566SW73S7mX9Z37f3mCXJKrP9w3xqYJxzHZWVlBQQEbNu2jYh+bpD/3CCYHn4CQTgItFrtpEmT9Hq90I0IxiMIw8PDdTpdaGgowzAxMTEWi0XA3nzPY2u4iHCeHD16dPXq1XPmzNmwYUNQUBAR/dzoukFcRDg9/MfAf1lG5MrKyjo7O2fMmEEICQ4Oxu+KueNPiR07dqwvT4n1T6KdJ5TSV1999fTp03l5eePGjXONi3ZudLtBRDs9/A0Olhkgi8WycOHCyspKu93++uuvL1iwQOiO/Iggp8T6J9HOkzNnzuTn5x84cCA2NtZsNpvNZn5ctHOj2w0i2unhd4TaOTnUOZ3Obdu2JSUljRgxYtmyZUajUeiOhOQxkQwGw/z58+Pi4pRKZXt7u1BdCcV9a4h2nmzYsKHbtxrRzo1uN4hop4e/wT5CAAAQNXw1CgAAooYgBAAAUUMQAgCAqCEIAQBA1BCEAAAgaghCAAAQNQQhAACIGoIQAABEDUEIAACihiAEAABRQxACAICoIQgBfKqqqkoikQjdBQD8G4IQ7k6/+MUvqqqqPAYPHjw4ffr0qKio1NTU0tLSPpYE8fXXX7suXQQAXoUghLsNy7Jbtmz5/vvvPcZPnTq1YMGCRx99dNeuXWPGjHn44Yc1Gk2vJaEolcrGxkZhewAQCVyGCe4q27dvf/HFF202GyGksrIyOTnZVVKpVPfdd9/mzZsJIZTSuXPnTp06dePGjT2XBl1VVdWECRN6fd1JJBKP/gHAS/CJEIaekydPBgUFFRUV8X++8cYYL4gJAAAFQUlEQVQbo0ePbmlpIYQsWrTo/PnzV65c8ViFUnrs2LElS5bwf0okksWLFx8/frznkgeJROL6utV9Vx9/+/r163Pnzo2Ojp48efLnn3/u3m1aWlp0dHRSUtLy5ct1Op2rVFFRsWDBgri4uLCwsJSUlLy8PNcDEUImTJjA36CUfvDBBxMmTJDJZFOmTNm1a5crR69cuTJv3jyFQhEdHT1nzpyu3wYDQK8QhDD0pKWlvfzyy5mZmSaT6fLly+vXr//kk09GjhxJCImJiUlJSUlJSfFYpaOjw2KxjBkzxjWSlJR069atnkv98qtf/WrWrFm7d++eOXPm0qVLv/76a0JISUnJrFmzYmJitm/fvnHjRpZl58+fzy9vs9nS09Nramr+8pe/7N27NzU1denSpUajkRCi1WoJISdPnuRvfPrpp1u2bPnTn/60f//+xx9/fMWKFdu3byeEcBw3d+7c4cOHf/DBBx9++GFwcPCyZcv62zYAEO9c+B7Au2w2289//vNly5ZNnjw5Ozu76wKEkMrKStefN27cIITY7XbXyOXLl6VSac+lHu6zsrLS9fLhb3/88ceuJf/whz+kpaVRSh955JHMzEz3O/ntb3/Lr2gwGNatW3fhwgV+vL293f3+3W9PnTr1+vXrrnvIzs5OTU2llDY0NBBCrl69yo+3trZ+8sknt9tiAHA7+EQIQ1JwcPBnn322d+9ejuPefPPNXpdXKBSEEJPJ5BoxGo38YA+lflEqla7bCxcuvHr1KiHk4sWLzzzzjPtiixcv5m/I5fJ169ZZrdb33nvv+eeff/DBB293z9XV1ffff7/kJ5s2baqpqSGExMbGZmZmPvjggyqVKjc312q1Pv300/1tGwAQhDBUNTc32+12rVZrMBh6XTg6Ojo0NLSurs41UldXFxsb23OpBwzD9FANCAhwOByEkKCgII+S+0mETz/99IoVK5qamp588smSkpLb3ZtMJisrK6t0U1xczD+KWq2+fv36Y489VlpaOnHixNWrV/fcNgB0hSCEIUmv1y9btuyNN94YN27cs88+S/twEObjjz/+xRdfuEb27ds3d+7cnktdufLv7NmzHqWDBw+6bn/11Vf8fsrJkyfv3LnTfTHXcTStra2fffbZiRMn1q9f/9hjj/XwFCZNmqTRaJKTk5OTk8ePH79ly5Zdu3YRQgwGw3PPPadQKLKysr788ssvv/zy/fff73k7AEA3hP5uFqDfnE7nU0899dBDDzkcjqqqqtDQ0B07dngsQ/5zHyGltKioKCQk5O233z558mR2dnZkZGRtbW2vJXeJiYmzZ88+fPjwzp07p0yZQv5zH+GIESM2btx44MCBF198USKRHDhwgFJ6+vRpiUSyaNGivLy8ffv2LVmyJD4+nl+RYRiZTLZy5cqSkpI9e/ZMmTIlMDBw165dNpuNUhoYGPjuu+/yexALCgoiIyP//ve/5+fn84fDHDx4kFLqcDhGjRq1ePHigoKCvXv3zp07d+bMmYO6pQFEAUEIQ49arZbJZDU1Nfyf77zzTkRExI0bN9yX6RqElNL8/Pzp06dHRkampaWdP3++jyWXY8eOJScnR0REzJ49u7a21iMIS0tLH3rooYiIiJ/97Geff/65a60TJ048/PDD0dHR991333PPPXf+/HnXivn5+UlJSREREenp6WVlZb///e8jIiJu3rxJKV21alV4ePiwYcP4JT/66KPk5GSZTDZt2rR//OMfrjs/e/bsjBkz+CUXLlxYX1/fv00JAJTihHqAO9XHc+QBwD9hHyEAAIja/wI/5sDk1Y65bgAAAABJRU5ErkJggg=="
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot([r0[2,:], r1[2,:], r2[2,:]]; xlabel=\"x100 updates\", ylabel=\"error\",\n",
    "    ylim=(0,0.15), yticks=0:0.01:0.15, labels=[\"MLP\",\"RNN\",\"biRNN\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3wUdf4/8PdnZrZvsptOEpIgEAhNOiJFBcUeECxfz3Z63oEF5SxYTkS901M45Zqeniin4u8sh6AiiqAUQRQQpUqvSUjPZjfbd2c+vz8WQkxCIMnuzib7ej74Y/P5THknJPvaz8xnZhjnnAAAAOKVoHYBAAAAakIQAgBAXEMQAgBAXEMQAgBAXEMQAgBAXEMQAgBAXEMQAgBAXEMQAgBA9NhstsLCwuTk5IkTJ9pstka9o0ePZifddddd0SkJQQgAANEzZ86cvLy80tLS3NzcuXPnNuzinO/Zs6e4uLiurq6uru5vf/tbdEpCEAIAQPQsWbJk+vTpOp1u+vTpixcvbthVXl7u9/snTZqUmZl5yy23OByO6JTE2nyLNUVRZFkObzWxJiAHyt2VXROyWlgmGAyKosgYa2EZHvAH7dWHNJm9LeEuMfZwzmVZliRJ7UJiiCzLjDFBwOfOUwKBgEajUbuKGKIoCudcFEW1C2mLm266yefzNWrU6/WvvvpqSkpKo3az2VxZWWkwGDweT0ZGRsO027p164MPPjhv3rzc3NwHHnjA7/e/9957Ea++PUEoy3IgEAhvNbGmxlv72aEVt/W9oYVlbDab0WjU6XQtLCNXl9W+8WS/cxbsuUaxajv5zV1lWa6pqUlLS1O7kBjicDhEUTSZTGoXEkPKy8vT09Nb/gQZV9xudyAQsFg65Iflzz//vOm46M0335w8efK0adMatZtMpurqar1e73a709LSXC5Xs9ssLS3t169fTU1NRCr+pbZ/bBdFsYN+eDl7WfouU4fc1vIyOp1Or9e3HISUlWf3eS9PcnxbnXTtOZ18WCDLcuhnonYhMcTv94uiiJ9JQ6FfEgRhPUVROu4vyZQpU5o2fvXVV80unJWVVVRUlJ+fX1JSkp2d3bDrxx9/9Hq9o0aNIiKtVnuG99Xw6eRvyu33wvf/OGw/1t6tMKbp2nOy5tCXJZ18OAgA0LLCwsIFCxZwzhcsWDBp0qRQ45o1a4jI5XJNnjx59+7dfr//T3/60zXXXBOdkhCEZ/DIefedY8lt/3a0OfnDA/u/KEIQAkBcmz179vbt23Nycnbt2jVr1qxQ47hx44hozJgxTz31VGFhYXZ2ts1mmzNnTnRKwoyGMzhiP1blqR6ROaSd29Hm5Ft+XK0x0Z5aXmDF4SAAiFNWq3XZsmWNGkOzVRhj99xzzz333BPlkjAiPAOtqLHqwnD6WpOT7z+2f0I2+7IYg0IAgBiCIDyDrglZGaYwTICUkjMyHv7nZV3ZlyVK+7cGAADhgiA8g6Aiv/DdPzi1exjHmGC2XpItfFvGPcFwVAYAAOGAIDwDSRCfv2gWo/Cc1UvU0LkpbH05jo4CAMQKBOGZfbD74x9Kt4ZlU1wOXpYtfFmMo6MAALECQXhmV/a4ZHCXAe3fjmyvLvvTHZd1xXwZAIAYgiA8M5nL28p3tX87oiWFBwODtLZyDy9yIQsBAGICgvDMfEGfw18Xlk1pc3rKxfsuzha+wi1mAABiA4LwzDJM6RfmjgrDxFEibU6+v+jAZbiaEAAgZiAIz8of1jxb6w3Dk7E0Ofn+ov2X5wgrSxQZUQgAEAMQhGfl+YueTNKH4f4y2pxegWP7uhioq4n9UIkkBABQH4LwrHxX8sO6ou/bvx3Rmpp088PE+WVd2Re4iAIAIAYgCM9KftI5/dMKwrIpfcFQYuzmnsL8PdyNW8wAAKgNQXhWLLrECldVGDc4MJmNymCv7sagEABAZQjCs+KT/RtKNod3m08PEeZul+sC4d0qAAC0Dp5HeFbMWtMd5/4qLJtSvO7q+U+l3feXfkns4izhlZ+Vxwbi4wiAmoLB4PDhw9WtgXPOOReEGH03mDVr1rXXXqt2FZGCIDxb8za9elWPCb1TerZzO4LeGKw6LtsqxKT0Z4YIo5YG7+ojWLVhqREA2kJRlJ07d27atEntQmLUc889V1UVznNDsQZBeLYeHHF3uDal6ZrvLzpgSErPt7CrcoR/7FJmD47Rj4EAcYIxNnjwYLWriFEpKSlqlxBZeP89W4drj649tiEsm9Lm9AwU7Q+9fnqI8PIuucYXlg0DAECrIQjPlllryk7oEpZNaXN7+Yv2hV53S2CT8oS/7pTDsmUAAGgtBOHZSjWmJGjNYdmUJifff2wf8RN3lnlysPCvn5UKT1i2DQAArYMgPFuM2L+3vhNUwjB0ExOSEi75Py6fuJw+18z+r7swD4NCAAA1IAhbYfbohyVBDMumEsZfxyRN/ZdPDhbf3KuUY1AIABB1CMJWWLRn6bqi7yKx5Uwj3dxTmLMNg0IAgGhDELbC1T0njM05P0Ibf2yg+PZ+pQRPrgcAiC4EYSt4Zd+3xWG75Nax/F3/kd31X3Yx0G96CS9sw91HAQCiCkHYCpyTzMN29JJpdK7vv2zY8uhA8b2DypE6DAoBgBhjN910U8OWm2++mTFW39vsKpIkSZKk0+lGjBixdevWaBTa8SEIWyFJbxnddYTCwxNUxhGXeLZ/y/2nrqVP1dPUAgwKAeCENWvW+Hwn3iL8fv+aNWvOuEowGAwGg2VlZRMnTpw6dWpk6+ssEISt89yGv1a6K8OyKTEhSdutwLP924aNjwwUlxxVDmFQCABEY8aMWblyZej1qlWrRo8efZYrJiUlzZgxY+fOnRErrVNBELbO7NEPZ5jSw7U104hLXZtWNmyxaunuPsKzP2FQCAA0ZcqUjz76KPR6yZIlkydPPssVq6qq5s2b16dPn4iV1qngptuts/H4llqv/bLu48OyNf2A822LXg5Wl0kpp27e9kB/Mf/DwF670NvSzDkAAIiOdw8oiw5H6djMrT3Ztec0Myy56qqrHnrooUAgIIri8uXL586de8ZN1Z877N+//zvvvBPmQjspBGHrFKTkCyxsw2gmSsm3PioYTA0bLVqa0V989idl4UXhuXgfANpgaCoza868WFj0tTb/qTchIWHo0KFr1qwxGo39+/e3WCxn3BQP0ySGuIIgbB2jxrircvegjAHh2qC+95CmjTP6CfkfBnbXCn1O8+cBAJHWx8pi4Q8wdHTUbDaf/XFRaC2cI2wlzrdV7Ir0TswaemCA+PSPOFMIEO8mTpz42WefffrppxMnTlS7lk4LQdg6GlHz6wE3hn2z9TfgrndvX2FdmbKtBkc5AOJacnJy3759MzMz09MbT9OTGlCltk4DQdhqr/y4YHtYB4Xc5yn746+539uw0STRzHPFZzAoBIhX9Wf7VqxYsXbt2kaNnPNgA416oVUQhK1275DfnJveL4wbZDqDpmtPz7b1jdrv7iNsruSbK/GbDQAQQQjCVjtqL/ri4Ffh3aZpxCWuTY23qRfpsYHCMz/ikRQAABGEIGy1RF1iQUp+eLep739+oPRwsLq0UfvvCoSfa+n7CgwKAQAiBUHYakl6i8AETuEMJyZKxiEXuTd/3ahdK9BjA4WntmBQCAAQKQjCtvho72e+oO/My7WGacQE95bVTdvv6CUccNDaUgwKAQAiAkHYFg+OuFsv6cO7TU3Xnum//2sz7QI9OViYhUEhQJypf6ZS6LFKw4YN27hxY33X008/3WjhM3bB6SAI22Lx3mUrDq8J+2YFU2Kz7bfmC5UeWo1BIUCcqb86wm6333jjjXfeeWd910svvbRjx45m12qhC5qFIGyLifmXX3rORVHbncjoycHCrB8wKASIU3q9/q677jp27Fh9y2OPPXbHHXfUX0HYUAtd0CwEYVu4A+6VR9ZEYsuKxxmsLGna/qseQl2AvizGoBAgHrnd7vnz548ff+q5N4888ogsyy+99FLThVvogmbhxjxtIQqiUTJGYsu+/duc33ySNr3xw1YERrMGCbN+kC/tKuF4P0AUeHdt9O79sWGLYExIvPyW0GvFXedY/m6jVcyjr5YyckKvnd9+Fiwvatirye5uOu+y0OtAySHXxi/ru/T9zmv2/vsNT+8JgrB+/anbbmg0mgULFlxwwQWTJk0qKCj4xY5O3wXNwoiwLUwa47DMQUEl/Mcq9f3OC5QdC1Y1vqCQiK7vLgQUWnYMg0KAaBBMiVJqVsN/YnJGfS8TpUa9UmoW0+rqFxATUxqvnpB8anWdoWGXYDQ3WwM/yePxvPjii9OnT2/YO3jw4BkzZtx5552y3Pi9qIUuaAojwjZ6ecsbN/aZnJ2QGd7NMlEyDrnQ/cPX9R88T3URPT30xKBQiw8wABGm7dZH2+20T3hnOoP5gkktrG4YcH4LvVJqZsurN6LX6+++++4nn3yyUfuTTz65ePHil19+uekqLXRBI3hDbaOHRtwT9hQMMY241L1pJTV389xJeUL3RHbfBnzEA4g7er3e4/E0atTpdAsWLJg1a1bT5VvogkYQhG30Q+nWxXuXRWLLmq49mMHkO9jM7GdG9M6F4oZy/tpuPJUCIO5kZGQ0bRw5cuS0adOaXb6FLmgIQdhGfVJ7XdZ9XIQ2bjr/ysDxw812mTW09FLxmR9l3GsGoHNr+kyl48ePN9v14osvNnw80+m64HQQhG2kl3Rby3dGaOPmMVe3cP6gWwJ75yLpptVykQu/3wAA7YUgbCNGwlFHUXhvvX32JmSz3/cXJq2Q3bhkFgCgfRCEbSQwdlPfaxmpdlHfzHOFAiubth4TZwAA2gVB2HZvbHt3U+mPZ16urdybVgZryltY4M2x4p5a/tedmDgDANB2CMK2++3AW0ZkNnMziHDxlxx0b1rZwgIGiT66RPzLdnk5br0GANBWCMK2O+YoWbRnaeS2bxp5uWvTimYvKKyXa2b/u1i6Y23woANZCADQFgjCtksxJI3IiuCIUJPZTTAk+A5sa3mx0RnsicHilK9kFybOAHQizT5HsOFzB/G0wnBBELadSWP0y75I3HH01C5GTHBt+uqMi03vK4xIY7eukTEqBOjcHn300frXeFphuCAI22X1sW89cuObHoWRcdh4767vFa/7jEv+a7RY5eXPb8XEGYDO7IUXXmjaiKcVthOCsF1+N/DWBE3zt40PC8GUmHDpTdzrOuOSGoE+vFh6bbfyGR5PARAOOyp/LndVuALuDSWbiGjtsQ1+2V9cd3x39T4iCj2RNFzLnM5TTz2VlpY2atSoAwcOhFqaPZKJpxW2E4KwXT7Z/8XKorUR3UXCRVNEa9rZLNnFQB9eLN7xTfDnWmQhQHu5Ax6fHFC44vA5iajOX6dwxS8H3AEPEdV6HWFc5nQURSktLb3ooovuv//+pr3sJJPJ9PDDDzc8ahp6JOGzzz67Z8+eRmu10BUdNputsLAwOTl54sSJNput2WV27txpMpmiVxOH9qmurvZ6vWpXccrb++TeHwZqfaoVEAwGy8vLVdt9TLLb7U6nU+0qYktpaamiKGpXcYLP59NoNGpX0RgRHTx4kHNeWVmZmJhY39joBefc4/HMmzdvyJAhjbqeeOKJUaNGhY6CnrGrBVOnTn3ttddaW//p1nr00Ufvvfder9d77733PvbYY00XqK2tHTp0aDTjCSPCdnEF3MuPrVK7il+4LV+4JJv936ogZs4AdA6CIGg0mhYWCD2tcO/evY3an3zySZvNdrqnFZ6uqw38fr+vCUVpfsrCkiVLpk+frtPppk+fvnjx4ka9nPPbb7/9scceC0thZ6ntD+b1er2nG9XGD78S0PjEsrIyrVYbub1wn5t/+m/hut/T2U10fjSPbtxkmfmNa2avM8+yCTtZlm02G56L3VBdXZ0oikajUe1CYkhFRYWiKDEydz8QCKhdQvPeeuut2bNnz5s3b9y4MzzrpoWnFU6YMKHp8i10NcU5t9vtpaWlp1ugoKCg6d61Wu2wYcOaLlxSUpKXl0dEeXl5Tbc5Z86cHj16XHfddWdTWLi0PQj1en1mZkSeTNuxMIklJVgTjAkR3Uu53211lut6DT7L5T+9kkZ8HByZk3BD92gP+mVZ1mq16enpUd5vLDOZTKIoRvWcR8xjjGVkZMRIEPr9frVLaJ5Go8nMzOzXr9/ChQvPuHALTytsdmpMC12NMMYsFksLb/h2u71p4+kehcg5D/2/c84bfWJevXr18uXLV65s6Y5akYBDo+216ODSCndVpPdiGjHB1eLt1hpJ0dFHl4j3fSfvqMERUoAOiXP+5JNPVlZWrlmzJicnp76x0Yt6HeVphVlZWUVFRURUUlKSnZ3dsOvrr79eu3atVqsNJSVjbP369VEoCUHYXrcX3Ng1ISvSezEOHefdtUnxOM9+lUEp7K8jxUkr5Spv5OoCAGidwsLCBQsWcM4XLFgwadKJB6+uWbOGiJ599tn6CSxExDkfM2ZMFEpCELaXwvkj3zzjlyN7aEUwJeryB3q2te7D0U09hGu7sV+tDgZxnT0AxIbZs2dv3749Jydn165ds2bNCjWe8SRoRCEI20tg7M9jZmnFCE6WCTGOmNDywyiaNWeEqBHo8c2YugIAMcFqtS5btqy4uPjTTz+1WCyhxqZHaKN5zBZBGAauoGvB9v9Gei+GvsMTL7+1tWsJjN69SPr4KH9yi4wLKgAAmkIQhoFJMl2QMzLiuxFEXa9BbVgvWUffTZQ2VfBLPg+WRfDGqAAAHRKCMAwkQUwxpOyo3K12IaeVqqflV0iXdxVGfBxcX4aBIQDAKQjC8Kj12e2+lu4ZGEaBkoNtWIsRPTpQeOtC8f9WyU//iKOkAAAnIAjD4xxL7qjs4a5AxO/kwv3eqvlP+Q5sb9vq47PY95PEFcX8mpWyzRfe0gAAOqS231kGGll2cGWiNuHC3FER3QvT6pNuuN/23ryMR15lOkMbtpBjYt9cLc3aIg9eEvzfxeLwtJi4tQeAujjnixYtUruKGHX48OEhQ4aoXUUEIQjDprDnZdHZkb7vCO3WdfalC6zX3du2LUgCvTBcPC9NKVwRfHygOKM/DgxAXNNoNC+++OL777+vYg2yLHPOJSkW35MTExO7d++udhURFIs/9I7rtZ/eGtN1ZP+0gkjvyDrlrvIX7tIPOF/fu+0f0yZ3E/olseu+lrdU8VfHiCb8LkC8YozNmDFjxowZKtbgdrsDgUD9dXUQTRgKhNNvzr0pCilIRILelHTDfbUf/pMH2nVHm14WtnGipBVp+Md4nC8AxCkEYThpRe2C7f+NwpQZItL3HZF8y8NM09472hgkemOs+NhAYdyy4IeHcCs2AIg7OBwWZiOzhulEXXT2pT2nX7g2dVu+MCCZXf+1/PVx/s9RohYfkAAgbuANL8x6JffYVrFT7SraYnAK+3GyVO2j0Z8Gj9ThMCkAxAsEYZhxrvxctZdT9IIkWFXqXLM4LJtK1ND/LhZv6SmMWhpcXowsBIC4gCAMM42oubX/De5A9O7pKZgT69Z+7DuwIyxbY0Qz+gvvj5d+u05+bDPu0w0AnR+CMPwO2A79J/IPo6gn6E1J10+3vfcS94UtfS/own64Rvq+go9ZGlxdijAEgM4MQRh+PZO6Tx/622juUd93hLZ7P/uyt8K4zS4GWnWl9OAA4a718pilwW9wq24A6KQQhBGxvnjjh7s/juYerdfe49m+IVwHSEMERtefI+y6VppaINyxVp7wRfDHKsQhAHQ2CMKIGJ45+LqCSdHco6A3JV1/b93qj8K+ZUmg2/KF3ddL158jFK6Qb/ha3mtHHAJA54EgjAidqP22eOPPVfuiuVN9v5Gpdz4ZoY1rBZpaIOy7QRqdwS76LHjD1/JBB+IQADoDBGGkZJozshO6RHuvghjRzZskmtFf2H+DZmgqG/lpcNp6+bgbcQgAHRuCMFJ6Jp1TUlfmDXrV2b0SwZulmTX06EBhz/WaJB31WxSctl4uj97VIgAAYYYgjKCdVbtdUbygsJ5z/dLaj/8d6b2k6OiF4eLeE3EYeGyzXNuuG4ADAKgDQRhBNxRMMmmM0d+vcdj4sM8gPZ10A70wXPxxsmTzUe//BZ7+Ua4LRGG3AABhgyCMIIUrT3zzXFCRo7zfSFxi37JcM/v3GHHNVdLPNuq3mL92UI+7lQJAR4EgjCCBCS+N/6MU4QkszdL3O097Tl/7529Hc6d9rOzDi8VPJ7CfHeLIT4N9FgUf/F5eUcK90f4kAADQCgjCyKpwVz3/3d9U2bX12ns8276NzgHShgYl0z8Gu8pu1iy6WMwwsL9sl7v8v8CEL4Jztil49i8AxCA8jzCyUg0pM4ZNU2XXgsGcdMP9itelyt6JqF8S65fEHh0ouIK06rjy2TF++ReKRqBLstkl2ezyrkKCRq3SAABOQRBGlsBYhbvqUO2R8Xljo793fd/h0d9pUyaJCnOFwlwiokN1fOlR/voe5Xfr5OFp7JIs4ZJsNjSVqV0jAMQvBGHEGSR9bmJXtauIFd0T2Iz+bEZ/oS5AX5Uoy4v5lK8UgdHlXdnlXdkl2YIJv5IAEF1414m4DFOayIQyV0UXU7paNQRKDklpWUyrV6uAphI0NLmbMLkbEdHPtfyLIv7yz8qta+SR6eyKHOGqHNbLgmEiAEQDgjAadlbtSdZbVQxC54ZlclVpym+fZhqtWjW0oK+V9bWyhwYI7iBtKOdLjykTvlBERhOy2SXZ7IqughlnEwEgYjBrNBouyh3dP62PzFW7jCDp2nsFs7X6zWd4IKbv/mKU6JJs9vfzxaM3Sl9dKfa1stf3KNn/PTHpFE+9AIBIQBBGyT+3zD9Ue1S13QtC8s0PC0Zz9YI/8WDHuPVL9wQ2o7+w8gqp5CbN/f2EQ3V8/DK5xwfBGd/JX5VwfwTvpQoA8QVBGCUzhk3LT+quZgWCkHTzTCZJNe+8QFG/2U17mDVUmCv8e4xYcpP06aVilpE9/aOcujBQuCL4+h6l2IVhIgC0C84RRs/T6+fOPG+6KncfDWGilHz7E45lbyk+j2Awq1VGe9Rfm1jhoS+Klc+L+OOb5QQNyzZRlpFlGSnbxDKNlG1kmUbqamK4VBEAzghBGD0PDr9bxRQMYaJkmfhbdWsIi3QD/Tpf+HU+BRWx2MVL3HTczY+7qMTNd9RQsUsp81Cxi3NOOWbWxUBdTwZklomyjCzbSF2MTK/Cze8AIOYgCKNHI2oW7102pfdVahfSqUgCdUtg3RKIqJnLLZwBKnLxUCiWuumwk68vp1K3UuKmUjcXGCXrWJKWknWUrGNJul+8SNKx5JMt1licbAsA4YEgjB6dqE0zpqhdxSnuH1b5Du1Muv4+Yp32ij2zhvpYWR8rNRuT7iDV+HiNj2y+ky/8VOPlx5xk81O1V7H5qcZHNV7uDFKSlpL1LFlH5ySw3hYqsLBeFtbLwnBpB0BHhyCMHoEJ52UNOWIv6mbJUbsWIiLDuaNc331Ru+Q16+S7OnEWtsAokVFiXU2hr1r6Ccicanxk8/FqHx1w8L21fPERvs+u7HPwZB3rbaFQKBZYWC8LdUtgYjz+OAE6KgRhVBXXlW4t3xkjQci0+pTfPVP1r8drP37dOlmdO4N3FCKjND2l6RkRnZ/+i5Sz+WiXjf9cyw/V8VXHlZ9tdNzNs4yseyL1tbJ+Sax7AksXWI9ElUoHgDNBEEZVd2ted2uezGWRxcQ8DUFvTL3nz1WvPG7/+HXLNVPVLqdDStLRmC5sTJdT6egJ0j4H32fn++y0voy/uVfZW6vjRLnmYK6Zckysq4nlminHzLoaKceMOTsAKkMQRtuKw6tdAffkXrEyZUbQm1LverbylUd1uzbq+52ndjmdgUGigclsYPKpaHQ4HH4uOgTjIQcdd/NSN31bzg8dUI676JiLawTKNLAsE3VPYJlGyjKy7gmseyLlmJgGF/oCRB6CMNouPWec2iU0JpgS0+5/SdAZ1C6kMzNIlGpi3Zub3Rqa1Frs4secVOTkq47zd1zKUSdVeHiKnswalqghq5YSNCxBQwlaStBQkvbU6wQNs2opUUNmDUvQECbvALQWglAFi/Ys7WbJGZY5SO1CThH0Kl/gGM+6GKiLgQ1r8lDGoEKVXqoL8LoA2XwnXoT+2fz8qJNOfqnY/eQIUJ2f1wXIHSSrjhI0zCSRSSKrjswSM2nIJFGSjkwSM0lk1pBVSyYNM0mUoCGLlkwSM2koESEKcQlBqIKLu4216CxqV9E8HvB7d/9gOHeU2oUASQJlGinz1AjyrKaiKpzsfnIEuCtIrgDZ/VQXeh2kWh85g9zmIleQbD5yBRVXkJwBqvWTK8BdQapr7ja0SbpmGkVGiZoT9WgEStVTqp6l6SnDQKl6lqqnND3LMFDohQ4nQSG2IQhVkKS3fnHwqzE5IxO0MXefM8VdZ/9kvuyoMY+5Wu1aoC0ERkk6StI1TM12Xcxh8zXTKHNyBE7c5dUvU5WXKr28wkuVHjpcxzdXUqVXqfBQpZeqvFwnUrqBhabdpuop3UDpekYencmmMMboZHjXq/Xz+hvI1gUoePIG6+4g+WTSiWSUSCuQSUMagcwSkwRK0JDAyKIlRmTVMjqZ31YtMUYWLZMYWXBXBDgNBKE6LPpEgcXiRAjRkpI2fU7ly48wxkyjY2VGD6io2REhEaXqG+XraePW7qdyD6/yUpWXV/mo3EPFLl5m1yT4iYgTnciwelbtqctac00knfxDMUqkE8krkydIPpncQQooZPPzoEL7AiRzcvhJIbL7Fc6p1k9EVOsnzsnu5wGFHAEKnUM1ScyipQQNmSQyaZhVG2okk8SSdKFGMp98rRXJJDGtcGLv4WX3k8Kp1s8VTmUOpshCtsCtWpagIcyTiiYEoTpGZY/YWbmnR1I3gxRDT40PEZPSU+/+c+XLj5AgmM6/Qu1yoMOzaMmiZb0s1DAsy8qcGRkmFt07OTgC5AqQK8hDZ1XrX9cFyBWkGh8/VEfOALmC5AoqNh+5guSXyRXkfuXEeFQvkkEinUBGiWlFMkmkEcisIYlRgoY1TPRQVHtl8sjcEySvfGJrziAPKOTwk8zJoiWBkVXLBEaJkhhUBJcs2/y8LkAagRI1lAR7Kv4AACAASURBVKBhFu2JqVKJWkrQUKKGLFpm0dLJL5lZc2J8TL88ZH26TzDQFIJQNXtr9mcndInBICQiKTUr7Z4XKl95lEka4/BL1C4HIDwST0wIavtx41CkheLNL5MrSAGFnAEKcqoLcFkhx8nzrKFDuHqRDKJgkEgvnhhfmiWmEShBc2qkG+J2uwOBgMVyYvaAO0h1AXL4uSNAtf4TL+oC5PCTzc8P15EjQHUB7vCT6+T4mH55yLr+mHaihkSBiMgoMZ1wqrbWcgYooNCPk6XOd5AZQaiaa3sX2ry1sXPHtUak9K5p984JVh1XuxCAGGKQyHDiXbNpgoZzdGuUyChRhqF1U6Wa5QiQrBDRiaEt0YnDy61l1lAowjsfBKGadlfvN2oMRLEYhEQkpXeV0ruqXQUAtEv9VTG/nEIFp+CErJpGZQ8fkNZnV9UetQs5s0DxQdeGz9WuAgAg/BCEKrN5ardX/Kx2FWfGdHrnt8tq3nle8brVrgUAIJwQhCpLNab8qu+U3dX7lFOXTsUiKS07/YG/CWZrxdy7/Uc6wBAWAOAsIQhjwvqi7z1Bj9pVnAGTNNYpd1smT6t64+m6rz+k2E5uAICzhMkyMeF3g26r9tg8QW+qIVntWs7AMGCUJuucui//y+UgkzrjBDIAiDMYEcaK70o2V7mr1a7irEgpmUk3PYQUBIDOAUEYK67ueWl+cvcj9iK1C2k1z44NOEwKAB0XgjCGHKo9uqVsm9pVtA73eepWfVT1+mzFaVe7FgCAtkAQxpD8pO7X9r66Yw0Kmc6Qft9cbV7v8rl3e/f+qHY5AACthiCMLUFFXrz3s6Aiq11Iawhi4uW3JN/6iO29eY7l75KinHkVAICYgSCMLZIgPjji7lqf3Rv0ql1L6+jyB6U/+A/fwR3+g9vVrgUAYpfNZissLExOTp44caLNZmvUu3z58r59+1qt1r59+65YsSI6JSEIY9HnB1eWuSrVrqLVxMTktHte0OYPUrsQAIhdc+bMycvLKy0tzc3NnTt3bsMuRVFuvvnmf/7znzU1NX/84x/vuOOO6JTU9usIZVkOBAJnXq6z8/l8oijysE6bvKHnpKAiF9lK0gwpYdxsdMiy7PP5vF5v4OB274ZlupFXaHsOpOg+di7WeL1eURRFMdzPde3IQr8kUX4eYSzzer2BQECn65BPEfzwww+DwcbPszh48OCQIUOaLrxkyZJPPvlEp9NNnz590qRJzz//fH1XMBhcuHDh+PHjnU6nTqezWq2RrfuktgehoigeT6zfDCUKPB4PY0wJ94mxLZXban2Oi7uODe9mo0CWZY/H4/F4eHo3yh/iXPYfCvilwRdJQy5mBpPa1akjFISCgAMwp3i93tDfjtqFxIpQEGq1HfJZfytXrvT7/Y0ajx9v/iFuJSUleXl5RBQaFzbs0mq1V155pdPpTExMZIytX78+QgU3wsI7lIlDNTU1JpMpQp/jyl2VGaa0SGw5cmRZrq6uTk9Pr2/xHdzhXPepb9/WhItvSLj4ehVrU4vD4RBF0WSK088BzSorK8vIyEAQ1mv0YN5OYNq0aUOGDJk2bVqjdpPJVF1drdfr3W53Wlqay+Vquq7L5fr73/++ZMmSzZs3R6FUfESNXXV+51s73lO7ijDQ9RiQcvsTGY++pus5QO1aAEBlWVlZRUVFRFRSUpKdnd2w68iRIzNnziQik8l055137t69OzolIQhjV4LW/OjI+6s8NTH+YIqzJFpStHkF9V/aP5lft2qR4naqWBIARF9hYeGCBQs45wsWLJg0aVKocc2aNUSUlZX15ptvrl27lnP+wQcfDB48ODolIQhj3cKdH1Z5OsY9SFvFMPjCQOmRsmdvt33w98Dxw2qXAwBRMnv27O3bt+fk5OzatWvWrFmhxnHjxhGRVqtdsmTJgw8+mJKS8v7778+fPz86JeHpE7HugeF3BZSgw1+XqE1Qu5Zw0ub2Sr75YcVZ6/zui6rXn5RSMpNu/L2Uln3mNQGgI7NarcuWLWvUWD9b5cILL9yyZUuUS0IQdgBfHlpl0SWMzTlf7ULCTzBbEyf8KvHiGzw7NgjmzjNNAAA6EARhB3B1z0uJqNZnt+o6aVQIomHgqQtFFGet/fN3zBdM0nTJU7EoAIgTOEfYMRx3lv1ne2eYQXpWREm0plW9+ofKfz3u3bURz3gCgIhCEHYMWeYuDwy/y+GvU7uQaBAM5sRLf9Vl9tum8y51fPnfsufudG34XO2iAKDTQhB2GJz4Sxv/5Q7Ey918mCgZh45Lf/Dvybc8onYtANCZIQg7DEbsmbGPCkyo8tSoXUtUabsVmEZdWf+lZ8cG376tKtYDAJ0MJst0MN+VbBYF8YLOOIP0bMly7ZLXiMh8wSTjsIuZpkPemxEAYgeCsIMZlzeGiL46snZwxrkphiS1y1GBYdBYw6Cxvv1bnWs/ti972zTyMvMF14iJ8fijAICwQBB2SFpRm6g1q12FmnT5g3T5g4JVpc71S+XaCgQhALQZzhF2SBfknB9Qgk+vn8spri8tkFIzrddM1eb2PvE159Vv/9n57WeK065qXQDQkWBE2FEZNYbfD59GRA5fXaKuU919re04Nw4d7/lpreOzt7R5vQ2DLzScO0owxPXQGQDOCEHYgVl1lh9Ktx6yH72hYJLatcQGQTD0H2noP5L7fd5dG90/rbUv+Xf6wy9LqZlqVwYAsQtB2LENyxw0LHPQjsqfM00ZqcYUtcuJFUyrMwy+wDD4Au7zMJ3hRKuieHZ9ry8YhommANAQzhF2BsfrykRBVLuKWHQqBYlkl9257tPSp26q+X9/8f68mctBFQsDgNiBIOwMLus+PkGbMOf7fwQVWe1aYpeYkJR2zwsZj8/X5vZ2rHz/+OPXlc+5i/t99QsEK0t4wK9ihQCgChwa7SQkQby533WSIPpkv07Eob/TEhOSzGMnmsdO5D5PsLqMaXUnOhS56o2n5eoywZQopWSKqZlSShfjoAukjBxV6wWAMNi9e/dtt912//3333rrrTNnznz11VeHDx/+zjvv5OTkEIKwM+makLWv5uCXh1fdN/R3atfSATCdQZN1zqmvBbHL4/OJc7m2MlhdFqwulavKeODUeNH+6RvByhIpJVPqkqfJ7q7pkodzjQAdxX333ZeRkXHFFVccO3bstddeW7Ro0V//+tcHHnhg0aJFhCDsZHol9+iZ1P24s0wraDB3pi0YE5PSxaR0Xc9zG/WYRl4eKDsWrDruO7jDue6TYEWxafTV1mumnuhWZMJpWoBYtXHjxjfeeCM1NXXu3LlXXHHF5Zdfbrfb77333lAvgrCzERjbVrGrV3IPBGF4SeldpfSu9V9yOcg9rvovneuWOla+r83ursnursnqrsnuLqV3ZSL+vgBigiRJjDEi+vbbby+99FIiMhgMfv+JOQH4Q+2Eruh+MSc+f9vCm/pea9IY1S6nc2KixMyW+i/NF15jGDQ2cPxQoOSQd9dGx8r3GGMZj70e6uXBAPd7yWRSqViAeHfeeectXbp0wIABq1atmj9/fiAQeP/99wcOHBjqRRB2TozYqOzhJo1R4VxgTO1y4oJoSREtKfo+w0983WAGL68sqX1ztjPBqunaU9u1p6ZrT21OT8FsVadQgPgzd+7cyy677N13373zzjvT09OnTp26atWqjz/+ONSLIOy0+qUWlLsq3tj27hOjHlS7lrjU4JShkNktafa7Oo/dX7Q/UHygbvWiQPGBLk8sEEyJoQVkR42YmKxSoW0XKD3i3rJacTmMQ8frevQnfOSCWHXuuecWFRWVl5dnZWUR0Zw5c1599VVRPPFHiiDszDJM6Y+cd58r4PYEvamGjvc+26kwJqVlS2nZNOSiRj08GKh4cTpX5PrBoqZrTyklJm8Lpyiy017/rA+5toqJopSWXfvRK9zvM464xDR8gpicrm6NAM2SJCk7Ozv0OinpF8+rwQX1nZxG1Gwo2XSo9ojahcBpMUmT+cf/Zsz8l3lsIZM07s1fV/5zpnf3D/UL+A7s8B3aGaw6rtb1/oGyY85vPql+85njT9zgWPaf+nZ9n2GJV9yWMP66jEdfS779D4rT7vzuc1UqBGjZ7t27hw8fvnDhQiKaOXOm2WweN25cUVFRqBcjws5vQreLiOjD3R93s+SOyBqidjnQvBOnGPuNbNrl3fOD/9Au2VEt26uZVi8mJhtHTEgYd22oV7ZXK3U2ITFFTLCG/eCka+OXjmVvM41Olz/QMPjCpP+bcbpTm9qcfG1OfsMWz47vRLNFe07f8JYE0Aa4jhCIiArzL5cEadPxH+1+RygaoaOwXH1H/WvF5ZAdNUzS1Lf4j+x2rHxPcdQorjohwSompUvJGcm3Phrq5cGAc+0SwWAmURJ0BqbTM0mjye4hGE88uovLwfrLPBSXw7d/G4miYcCoUIuu57lpM+ZJKV3aULbists/+w9xxTRignH4JaIlbNfzcDnIGMOFm3D2cB0hEBEZJD0RFaTky1wpc1X8UPrT1T0vU7soaDXBlFg/xSbEMHCMYeAYIuJyUKmzybZKxeM81a0oitsZrC6jYEDxe7nPw4PBxEtv1OUPCvXb/t9f3D+uZToDE0VSuLZHf9Owi+vXbs+pStPIy00jL/cf2ePatKJ8zl3avN6JV/660ajxdOSaikDZ0WDV8WB1WbCqVK4+LmV2S/n1H0K93l2bat7+s2C2iJZU0ZIiJqWJicnmcdfiwk04HVxHCKeEHuFb7bH1TS3wy/6vj667ovvFZ1wLOgQmSqI1TbSm/aJRq7MU/qaFtZJvezz5tse5z8ODAcFgCvswS9utQNutwDr5Ls/2bwWtvlGv4nXJVaXBqtJg1XFd78HanF6hdue3nwVKDkqpmWJqlq7nACk1q2EkG84dlf2XT+U6m1xbKdtr5NpK2V7N6itXlOOzfyUmJInWNDExWUxKEy0pul6DYmL+kaIoXpficXKPW/G5uc+jeN26/IFiwom5G+4f1yhOO9PqBb2R6Y2CTi8kprR6RM654nEpHif3ukRLqnDyglfPju/8h3cpXrfiruNet5iUrus5wDh0fHi/xdiE6wihsRRDUoohyea1W3QJRLShZPOo7OFnXAs6MaYzNHxkVfi3r9Eah4479bWiyAv/VFpbyYMBKTVTSsmUUjMZO5XBLYc3EZEghM6qNtvV5Q9vyPZq2VYpO2rk2kr/0b1SRm59ENYuetl3cKeQkCQmJolmq2BN1WTk6vsMO/tvh8tB7vMqHif3e7nPIyam1M+V9e3f6t3zI/e5FW8o5zyCwZjym9mh3mBNecW8+wRDAtMbBZ2B6Q2C3qjt2pNOBqHicQarjnOfV/G5udet+Lz63oMTL78l1Ove/FXtkteY1sB0ekFrYAaToDOk3PHEiY8vnJf96XbFXad43YLRLOhNzGBKvOxmw7mjT/43MMFkkVKzBIOZ6Q3BmnLu9dR/U4HSI55t6zVZ3TXZ50jJXTrZxTAtX0fIOOfq1tfR1dTUmEwmnU535kVjkifo/Xjf57/qO+Vg7ZHu1jxG7f3tl2W5uro6PR1z6E9xOByiKJpwZ5l6nJfu+ik975z6kVA0KW6nbK9S6myywyY7a5XaKtGaar5oSqjXu2eL7b15QkKSmJgsmBJ4wM89ruQ7nhD0J/77Sp+6Wa6rFfQGwWBmWj3T6c1jJ9XHvHfPlkDJQaY7lXOC2arJ7HbGqtxudyAQsFgsLS3EueJxcZ9b8Xu538s9LsXnqT+hS0RybSXTG+tLbZVgZYl701f+44cCxw9zj1OTdY4mq7t1yt0ktPHigmnTpg0ZMmTatGltWz3sgsFg6DpCxpjNZktMTMR1hHCCQdL/qu8UhfPFez97aMS9dr/dqmvxTxGg/RhjqVlq3VtHMJoFo5lOE066/IHpD/5DqbPJjhrF5WAaHdMbmXTqSSNdZr/dwslIfcFQfcHQsNd8AmOC0UxG8+mOXzc6MN4qUlp24lW/Dr1W3M5AycFgZUl9CnKfp+b9v2qze5jHTozowYPIwXWEcAYCYzPPmy4w9vfNr7sC7oCCp7dDnGKiJFpSNF176vuOMA6/xDBorL5gaMNpuvEwJUcwmnX5A02jrjzVxJih/0jF46QGP4qOZenSpWPHjk1NTU1OTh47duyyZcvquxCE8AtPjZlp0hjnbfpXSV2p2rUAQKxgWr1x6HhL4Z0d9HPA//73vylTpowdO/bjjz8OJeKkSZMWL14c6u2Q3xJE2sPn3SsyceHOD89N7zcwvZ/a5QAAtMuf//znRx555Lnnngt9OXr0aEVRnnvuuSlTphBGhNAskYlENKX31QPS+qwv3ri+eKPaFQEAtN2+ffvGjBnTsOXCCy/cu3dv6DWCEE7LpDEKTChIyR+Q1ueYo3jV0XVqVwQA0BZ5eXm7du1q2LJz5868vLzQawQhnEGqIdmiS9QImm6WXE/Qu/bYBrUrAgBonalTpz7zzDMLFy6sqampqalZuHDhH//4x9/97nehXpwjhLOSac4gojJXhSRIRLStYhfOHQJAR3H//fcHg8EHHnjgtttuI6KUlJTZs2fff//9oV4EIbRCF1N6F1N6rc++o3L3wPR+5a7KDFPbr1sCAIgOQRAefvjhhx56qLKykojS0tJYg1vn4NAotJpVZ7ml33UBOfDvn94iIp+szkPyAABahTGWnp6enp7OfnkDOYwIoY00omb2mJmc+B/WPvvCRbM1An6XACCG7Nmz54zLFBQUEIIQ2okR+8u4ZwTGnl4/98Hhd4eebgEAoLo+ffqccZnQ3bZxaBTaS2CMiGaeNz1Rl/D2jg+OOUrUrggAgPhZCC2JIITwMGmMRDQub0zXhKxvSr/bXPqTzOUtZdvUrgsA4AwQhBBOuYnZAmPnJvftndLT6XcdrD1CRE+tm+P0uw7bj+2tPkBEQUVWuUoAgAYQhBB+Vp0lUZtg0SXeUDCJiB4+716z1uQJeGUu+2T/zNVPEdGKw6tt3lpv0OcKuNWuFwDiGoIQIi5Bayaivqm9+qb21onal8b/iYjMWpNBMuyq2vP5wa+I6JUfFxCR0+/ihCdFA0BUIQgh2kKTa0Zlj9BLuqFdBl5fMJETH509nIje2PbuMXtxlafmsP2Y2mUCQLxAEIL6GLFBGQOI6PfDp+VZco7XlZa7KoKK/N+fP1K7NADo/BCEEHPOTe83MmtYQAn0Tu5JRE+vnxs6lYhZNgAQCQhCiFEGST+0y0Aium/o7/SSbtmBFT+Wb3MF3Dsqf1a7NADoVBCEEOtSDElEdEOfa0ZkDqn21JS5Kohowfb/ql0XAHQSCELoSHITu07odpHClX6pvYlo3qZXS53lNR7bcWeZ2qUBQEeFIISOR2DCeVlDieg3A2/qYk4/Vleyv+YQJ/74mj8R0ebSn6o9tqAiB5Sg2pUCQAeAIIQOzKqzMGKD0vtfmDuKiB4ZeR8RuQJugbF9NQfe3PYuEc3ftpCIyl0VDn+dutUCQGxCEEInwYgl6a1EdFHu6CS9tW9q77sG305EA9P7EdHm0q3FjtJyV2Xoyv11Rd8FlKDMMQ0VABCE0NmNyBxCRFf3vLRvaq9kvfX/+lxDRCXOMkkQlx1YufzQKpnLH+z+mIiqPDWIRoA4hCCEOKIRNamGZCK6sc9kRmxi/uWXnjMuqMg5idlE9PaO9+v8zu0Vu74+8g0Rba/YpXK5ABAVCEKIawJjOlE7Kns4ET004h6rzpKTmN0/rUDhfF3xRiJ67ae3jtiLqjw1oUdKYQIOQOeDIAT4hSS9NcOULjB275DfENFN/a7NTsj0BX1+OUBE9698nIiW7Fu2ufQnp98Vupzx84NfuQLuCndV6DlTe6r3ExFuhQOdCSful/2vb31H7UIiAkEI0JJEbYJGkLITMs/PHkZEr172FyK6uNsFA9L6akXN6K4jiChBaxKZaPc5bL5aIlp2cCURfbp/+baKnRXuquc2zCOiD/Z94gq4/bJf4Xi8BnQYpc5yn+zfUrbt7R3va0Vt6LKldrLZbIWFhcnJyRMnTrTZbI16P/nkk/79+1ut1gsuuGDfvn3t393ZQBACtFqiNkEv6bSiNnQ31LE55+slXX5S95FZw4jooRH3ENGv+k4Z2mVgmjEl9GXvpB4mjXHF4TUrD68OKvLXR9ep+y0AtCz0fLR1Rd8VOYoHpve/tf8NdHIOdjvNmTMnLy+vtLQ0Nzd37ty5DbuOHTt2yy23zJ8/v7S0dOLEiXfccUf7d3c2EIQAEcSI6SU9EQ1K609EV/e89LLu490Bt6zIRPTH9X/xBn2V7iqbt1blQkFtftlf6alSsQBOXOZyuavynZ0fEFFQCSpcuaHPNT2TukuCKDIxXDtasmTJ9OnTdTrd9OnTFy9e3LDr0KFDN9544/nnn28wGH7961/v3bs3XDttGeNtPVDj9/udTmd4q+mIbDab0WjU6XRqFxIrZFm22WypqalqFxJD6urqRFE0Go2N2mu8tmR90rrSjVZt4jmJuSuL1k7ufqVP9utE7dls1q8EJCZ6ZV+VtybXnL2yaM2EnIt+qtzBiQ9OG/DlsdWX546v9tYkahI0oiYC31a7VFRUpKWlMcbULkR9u237dKLOTMb/Hfr03kF3/mvnf24vuLHKW+OTffmW7jafPUlniWgBB+yHe1i6fXlstUHSj80cebSu+JzE3FZt4fbbb/f5fI0aDx48OHPmzGnTpjVqN5vNlZWVBoPB4/FkZGQ4HI6mG5Rlefr06YIgvPLKK62qpG2ktq8pSSaTKYyldFA+n89kMiEI68myHPqZqF1IDJFlWRTFpj+TUMvlPccTkSvgHpY92GQyPfPNi8+NeeJQ7RGjxtjFmFbjq80wpn1bsmlk1tD9tkNOv2tYl0F/+v6lJ0c+9MHO/47KGp6st+4s39Mno5dBb9Qb9b0z8jnnOr3eYkw0mUzvH/r4qu4TSn0VO6v2XN9r4meHVlzd/dJan10n6gySXoWfxUlGo9FkMsVtENq8tUl667wfXv3dubemJKQoXM7Qpk/r/2uTyXRL/+vTTKm1ioM4GYzGudtefmbUo+/tXjy260iTxnjUUTwovX+d35mgNbezhoAcWHF0zVXdJ2w/ujsvJXdywVUCE4iov7lPazf129/+VpYbTw2bP39+swtzzkP/75zzpmsR0VdfffXII49ceumlzz77bGsraSMO7VNdXe31etWuIoYEg8Hy8nK1q4gtdrvd6XS2apWNx7ccqT1W6a7+15YFnPNFe5Z6Ap7jdWWHao9yzsucrfsJewJem7eWc750/5ec88V7P9tWvrPCVfm3zf/mnH9x8Gtv0Ofw1YWWiY7S0lJFUaK2uxhR5CjZU7XfH/TP/uYFznmlu1pW5FCXy+WqrT3tz7+krtQb9BU5StYVfc85f/DrJznnXx5ata18p9PvWn7wa875R3s+c/pde6r2rz66nnP+0sZ/cc4/3vf59oqfq901L258hXP+1Lo5NZ7adUXfv7X9Pc55aMUImTp16muvvda0vWfPnvv27eOc79u3Lz8/v2GXoiiPPfbY2LFj9+7dG7nCmkIQtheCsBEEYVNtCMIoCMjBKncN53zZgZWyoqwr+n7VkXWyoty1/CHO+bs7/3fUXlTuqvzq8FrO+YbizQpXar12p98Vlr2fZRDW+ZxV7hqFK+/s+IBz/tdNr/1ctfeg7cjfN7/OOX9i7XOhykNv/fM2vco5X7r/y2LH8XJXZehd/uN9nytc2V9zaH/NQc75Fwe/5pxvK99p89bWeu1by3dyzr85tkHhSkldaUldKef8h9KtnPM91fvtPocv6Kt217TnO/UFfZzz/+76qKSu9Ki9+NviTc0u1nIQNqvUWW73OTwBT+jb33h8iyfgqXLXhD4t/Vy1l3Ne5a5x+l2yIoe+C3fAI0fr88fpgvCBBx547LHHQpn38MMPhxpXr17NOV+/fn3v3r1tNlvdSdEpFZNlAOKUJIihZz1e2eMSgbExXc8blzdGYOyVS+cS0bi8MWnGVI0gpRiTiWh39T5G7IfSrVvKtvll/wNfzyKid3Z+UOaqOO4s+7Z4ExFtLd9BRO6Ap823HVC44pP9G49vIaKXt7xR67Pvqzm46fgWRqybJZeI7h82tU9KrzxL16mDbiOi3w+bJjDWP61gcMYAIrq8+3giyk/ubtElakVNpjmDiDSChhELKIFQVe6Ah4jKXJW+oM8b9Ja7KonoqL2YEStzVpQ6y4noh9KtRLSv5qDT7ypxln2yfzkRzfjqCU782+JNW8q2KZyHJlUesRd5gl6Zy01vzufw1+23HSKih1Y9RURjc0ZadIm5idmhuzeERRdTeqI2QS/pL8odTUQjMofoJX2KIekcSy4R9UnpRUQphiSTxigwIdmQREQGSS+ofTh69uzZ27dvz8nJ2bVr16xZs0KN48aNI6I1a9bs3bs3KSkp4aQo1RSdvO3EMCJsBCPCpmJzRNgeoVHOkdpj3qCvzFkRGleFjuIu3vvZ+qKNdp/j+Q1/45x/uPsTu9dR6iwPDVCO2YtDWygtLfUFfKXOcs75wp3/U7jy8b7Plx/8OqgE/7vrI8758boyvxxQ6ftrhifg4ZyXOcsr3dX+oP/zg19xzhfu+LDSVbW1fOfrP73NOX9m3VzO+Xclm/fVHKx0VS3as5RzrvCzGoG1YUQY4043IoxBbZ81CiE1NTWYLNOQLMvV1dXp6elqFxJDHA5Hs5NlOjGFc7vPnqS3/lS+o19q7+K60kp31XlZQ5/9dt6s0Q++se3dbpqu/XP7fnpg+dRBt20o2Twic4jAWGiyRsdVUleanZC5t+aAVWfJMKW1al232x0IBCyWyE4QjaZp06YNGTKk6azRGIQgbC8EYSMIwqbiMAhbpnBeWnY8q0tW3M4abQpBqKKO/fkLADoigbEwXqAN0E4IQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIN4jDAwAADvlJREFUQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiGsIQgAAiB6bzVZYWJicnDxx4kSbzdZ0AVmWCwoKolkSghAAAKJnzpw5eXl5paWlubm5c+fObdT797//fdSoUXv37o1mSVJ7VlYUJVx1dFzKSWoXEivwA2lKURTGGH4mDYV+SRhjahcSKzr0H47b7eacN2oMBoPNLrxkyZJPPvlEp9NNnz590qRJzz//fMPec889t0ePHoWFhZGqtTltD0Kfz1dbWxvGUjqo2tpaj8ej1WrVLiRWyLJss9nwBtdQXV2dKIput1vtQmJIdXW1IAj4Pann8XgCgYDf71e7kLbo27evx+Np1ChJ0ogRI5ouXFJSkpeXR0ShcWGj3nHjxkWoyBa0PQh1Ol1GRkYYS+mgNBqNyWTS6XRqFxIrZFmWJCk9PV3tQmKIwWAQRdFkMqldSAzhnGdkZCAI67nd7kAgYLFY1C6kLaqrq5s2Tps2rdmFOeeh/3fOuSzLka3s7OAcIQAARE9WVlZRURERlZSUZGdnq10OEYIQAACiqbCwcMGCBZzzBQsWTJo0KdS4Zs0aFUtCEAIAQPTMnj17+/btOTk5u3btmjVrVqhRlVOD9do1axQAAKBVrFbrsmXLGjU2mnTadA5qRGFECAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1BCAAAcQ1B2F41NTU+n0/tKmKILMuVlZVqVxFb6urqXC6X2lXElrKyMrVLiC1ut9tut6tdRZxCELbX3XffvXr1arWriCHHjh274oor1K4itsyZM+eNN95Qu4rYMnLkSI/Ho3YVMWTRokV/+MMf1K4iTiEIAQAgriEIAQAgemw2W2FhYXJy8sSJE202W6t6IwRBCAAA0TNnzpy8vLzS0tLc3Ny5c+e2qjdCpDavefjw4RUrVoSxlA6quLh4+fLlOPNfr7Ky0ul0/vvf/1a7kBiyc+fOY8eO4WfSkCzLb775plarVbuQWLFu3boDBw500F+SjRs3yrLcqHHLli1DhgxpuvCSJUs++eQTnU43ffr0SZMmPf/882ffGyGMc962Nbds2fLPf/4zvNV0RB6PR6PRSFLbP1J0Mpxzt9ttMpnULiSG+Hw+xhje9BtyOp1ms1ntKmJIIBCQZVmv16tdSFt89913iqI0ajQYDBs2bGj6v2w2mysrKw0Gg8fjycjIcDgcZ98bIW1/+x46dOhbb70VvkoAAKDz45wzxkIvmo4jW+6NEJwjBACA6MnKyioqKiKikpKS7OzsVvVGCIIQAACip7CwcMGCBZzzBQsWTJo0KdS4Zs2aFnojre3nCAEAAFqrtrb25ptv3rZt25AhQxYuXGixWIiIsRNh1GxvpCEIAQAgruHQaLuMHj2anXTXXXepXY6aZFkuKCho2KLKhbGxo+kPJM5/Wz755JP+/ftbrdYLLvj/7d1ZTBNbGADgU0DAUqyyqIGohGotgkGWYMJSQIQiBln0iaXagEQjCVojib5gEAwxihCCgkSkKJugFcqDDwKCIIkgSoi0yCJBsAKBsmPLMvdhcufWglWJ1wHm/55OzzkzPfz5y5/O6bTcjx8/4p0UT5JlY0LxPCEFFMKVwzBMJpP19/dPTk5OTk6mpaWRvSLSpKenu7q6dnR0qHeScmPsKrE0IBTPlr6+voiIiJycHLlcfuzYMYFAgPdTOUmWjQnF84Q0GFgpuVzOYDCcnJwYDEZQUNDg4CDZKyJNdXW1RCLRSCc2my2VSjEMk0qlbDabpKWRY2lAKJ4tNTU10dHReHtoaMjU1BRvUzlJlo0JxfOELLBHuHLv378XCoWpqak7d+68cOGCSqUqKioie1FkIra7caTcGLuqqAcEsgW3sLAQGxuro6OTmZmJIEkQQt/HBPKEFFAI/wy5XG5razs6Okr2QsikUQiNjIxGRkYMDQ1nZmbMzc0p+IN8GgEhUDZbXrx4ER8f7+fnl5SUhH8ZEyTJ0pgQKJsnfx98MdjKtbS0fPv2zdXVFSGkr69vYGBA9opWF/zG2D179vzNG2NXLYpnC4ZhV65caWhoKC4uZrPZRD+Vk2TZmFA8T8gCH5ZZuenp6ZCQEKlUqlKprl27FhwcTPaKVhdSboxdtSieLa9fvxaLxRUVFRYWFlNTU1NTU3g/lZNk2ZhQPE9IQ9bm5DqwuLiYmZnJYrHMzMz4fP74+DjZKyKZRjopFIqAgABLS8vAwMCxsTGyVkUi9YBQPFuSkpKW/c9D5SRZNiYUzxOywB4hAAAASoNLowAAACgNCiEAAABKg0IIAACA0qAQAgAAoDQohAAAACgNCiEAAABKg0IIAACA0qAQAgAAoDQohAAAACgNCiEAAABKg0IIAACA0qAQAkAOmUxGo9HIXgUAAAohWO8OHjwok8k0OiUSibOz86ZNm9zd3Zuamn5xiBSVlZXEjxYBAP4PUAjBujU3N5eRkfHmzRuN/levXgUHBx8+fFgkEllbW3t4eAwMDPx0iCyBgYH9/f3krgGA9Q1+hgmsT9nZ2XFxcUqlEiEklUo5HA4xFBQUZGVllZ6ejhDCMIzH4zk6OqakpGgf+uNkMpmNjc1PX4A0Gk1j/QCAPwveEYI1rK6uTk9Pr6amBn+YnJy8ffv2oaEhhFBoaGhzc3NbW5vGIRiGVVVVhYeH4w9pNFpYWFh1dbX2IQ00Go243Kq+1Ye3u7q6eDwek8m0t7cvLCxUXy2Xy2UymSwW69SpUyMjI8TQhw8fgoODLS0tN27caGdnV1xcTDwRQsjGxgZvYBiWlZVlY2NDp9MdHBxEIhFRR9va2o4cOWJiYsJkMv38/JZeDQYA/AgUQrCGcbncS5cuCQSCiYmJ1tbWxMTE/Pz8rVu3IoTMzc3t7Ozs7Ow0DpmcnJyenra2tiZ6WCzWly9ftA/9lqNHj3p5eT18+NDT0zMiIqKyshIhVF9f7+XlZW5unp2dnZKSMjc3FxAQgM9XKpXe3t6dnZ1Xr14tKSlxd3ePiIgYHx9HCMnlcoRQXV0d3nj06FFGRsbly5efPn3q7+8fFRWVnZ2NEFpYWODxeKampllZWffu3dPX1+fz+b+7bACo6//54XsA/hKlUnngwAE+n29vby8UCpdOQAhJpVLiYXd3N0JIpVIRPa2trRs2bNA+pOWcUqmUeB3h7QcPHhAzz58/z+VyMQw7dOiQQCBQP8np06fxAxUKRUJCwtu3b/H+sbEx9fOrtx0dHbu6uogzCIVCd3d3DMM+f/6MEGpvb8f7h4eH8/PzfxQxAIAGeEcI1jZ9ff2CgoKSkpKFhYXr16//dL6JiQlCaGJigugZHx/HO7UM/ZbAwECiHRIS0t7ejhB69+7dyZMn1aeFhYXhjc2bNyckJMzOzt65c+fMmTMuLi4/OnNHR8fu3btp/0pNTe3s7EQIWVhYCAQCFxeXoKCgGzduzM7ORkZG/u6yAaAsKIRgzRscHFSpVHK5XKFQ/HQyk8k0NDTs7e0lenp7ey0sLLQPaTEzM6NlVEdHZ35+HiGkp6enMaR+E2FkZGRUVNTXr1+PHz9eX1//o7PR6fSWlhapmpcvX+LPkpub29XV5evr29TUtG/fvvj4eO3LBgAQoBCCtW10dJTP5ycnJ7PZ7OjoaOwXPoTp7+//+PFjoqesrIzH42kfWoqof42NjRpDEomEaD979gzfp7S3t8/Ly1OfRnyOZnh4uKCgoLa2NjEx0dfXV8ufYGtrOzAwwOFwOBzO3r17MzIyRCIRQkihUMTExJiYmMTGxpaWlpaWlt69e1d7HAAA/yH72iwAK7e4uHjixAk3N7f5+XmZTGZoaJiTk6MxB32/R4hhWE1NjYGBwc2bN+vq6oRCobGxcU9Pz0+H1O3atcvHx+f58+d5eXkODg7o+z1CMzOzlJSUioqKuLg4Go1WUVGBYVhDQwONRgsNDS0uLi4rKwsPD9+xYwd+4MzMDJ1OP3fuXH19fVFRkYODg66urkgkUiqVGIbp6uqmpaXhO4jl5eXGxsa3b98Wi8X4x2EkEgmGYfPz89u2bQsLCysvLy8pKeHxeJ6enn800gCsZ1AIwRqWm5tLp9M7Ozvxh7du3WIwGN3d3epzlhZCDMPEYrGzs7OxsTGXy21ubv7FIUJVVRWHw2EwGD4+Pj09PRqFsKmpyc3NjcFg7N+/v7CwkDiqtrbWw8ODyWRaWVnFxMQ0NzcTB4rFYhaLxWAwvL29W1pazp49y2AwPn36hGHYxYsXjYyMtmzZgs+8f/8+h8Oh0+lOTk5PnjwhTt7Y2Ojq6orPDAkJ6evr+71QAkBhcEM9AH/ML94jDwBYVWCPEAAAAKX9A+U37EnzawVLAAAAAElFTkSuQmCC"
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot([r0[1,:], r1[1,:], r2[1,:]]; xlabel=\"x100 updates\", ylabel=\"loss\",\n",
    "    ylim=(0,.5), yticks=0:0.1:.5, labels=[\"MLP\",\"RNN\",\"biRNN\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Playground\n",
    "Below, you can type and tag your own sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tag (generic function with 1 method)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wdict=Dict{String,UInt16}(); for (i,w) in enumerate(words); wdict[w]=i; end\n",
    "unk = UInt16(length(words))\n",
    "wid(w) = get(wdict,w,unk)\n",
    "function tag(tagger,s::String)\n",
    "    w = permutedims(split(s))\n",
    "    t = tags[(x->x[1]).(argmax(Array(tagger(wid.(w))),dims=1))]\n",
    "    vcat(w,t)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdin> colorless green ideas sleep furiously\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2×5 Array{AbstractString,2}:\n",
       " \"colorless\"  \"green\"  \"ideas\"  \"sleep\"  \"furiously\"\n",
       " \"jj\"         \"jj\"     \"nns\"    \"vb\"     \"rb\"       "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag(t2,readline())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "julia.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Julia 1.0.3",
   "language": "julia",
   "name": "julia-1.0"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.0.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
